From treumann@XXXXXXXXXX Wed Jan 9 15:27:29 2008 Delivered-To: mpifrm-mpi-21-outgoing@XXXXXXXXXXXXXXXXXXXXXXX X-Original-To: mpifrm-mpi-21@XXXXXXXXXXXXXXXXXXXXXXX Delivered-To: mpifrm-mpi-21@XXXXXXXXXXXXXXXXXXXXXXX In-Reply-To: <6B68D01C00C9994A8E150183E62A119E6F9B942C13@XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> Subject: RE: [mpi-21] Proposal EH2: add const keyword to the C bindings To: mpi-21@XXXXXXXXXXXXX X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Richard Treumann Date: Wed, 9 Jan 2008 16:26:58 -0500 X-MIMETrack: Serialize by Router on D01ML064/01/M/IBM(Release 7.0.2FP2 IGS702FP2HF5|November 8, 2007) at 01/09/2008 16:27:00 MIME-Version: 1.0 Content-type: multipart/alternative; Boundary="0__=0ABBF958DFE3719C8f9e8a93df938690918c0ABBF958DFE3719C" Content-Disposition: inline X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at mailbouncer.mcs.anl.gov Sender: owner-mpi-21@XXXXXXXXXXXXX Precedence: bulk Reply-To: mpi-21@XXXXXXXXXXXXX X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at mailbouncer.mcs.anl.gov X-Spam-Status: No, hits=-0.8 tagged_above=-1.0 required=5.3 tests=AWL, BAYES_00, HTML_20_30, HTML_MESSAGE X-Spam-Level: --0__=0ABBF958DFE3719C8f9e8a93df938690918c0ABBF958DFE3719C Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: quoted-printable An obvious question to me at least is: "Who is the user?" The person who must write an MPI application and get it right it seems = to be the "user" Erez is thinking of. The organization that buys and operates a parallel machine to solve problems is also a "user". The implementor of an MPI offering serves both kinds of user. Preservin= g specific freedoms for the implementor is not about making the implement= or's job easy. Let's pretend an MPI implementator who can assume the user will obey th= e standard could make MPI_Isend 5 times as fast as one that must protect= some user who ignores the standard. In this case, the second class of "user" would probably say. "Let the MPI programmer learn the rules. I w= ant the machine performance." Now let's pretend everyone agrees the performance advantage of being ab= le to assume the user will always honor the rule will never exceed 0.00001= % on any machine, now or in the future. In this case both types of user an= d the implementor will all agree that the rule should be removed and the application writer relieved of the need to remember an "unnatural" rule= . I said *pretend* in both cases because I know the real situation is far= less clear than the absurd performance numbers I used. My point is that any proposed simplification of the standard will not simply be a trade off between 1) making life as easy as possible for people who write MPI libraries or 2) or making it as easy as possible for people who write MPI applicatio= ns. I think it is likely the forum will conclude that the send buffer acces= s rules should be relaxed but there should be more to the discussion than= "Do you care about users or don't you?" Dick Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 owner-mpi-21@mpi-forum.org wrote on 01/09/2008 02:58:21 PM: > This issue is related to the "send buffer access". I say "wait, > let's make it right for the users by restricting the implementers". > > The trick you described is legal with the current standard, but is > so surprisingly wrong for many of the users. > > > -----Original Message----- > From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org] > On Behalf Of Supalov, Alexander > Sent: Wednesday, January 09, 2008 11:43 AM > To: mpi-21@mpi-forum.org > Subject: RE: [mpi-21] Proposal EH2: add const keyword to the C bindin= gs > > Wait, we're talking about standard here. The program you showed is pl= ain > incorrect. The trick I described is legal. All according to the > standard. Let's keep our bearings straight: we don't want to make bro= ken > programs work by reducing the freedom of implementors. > > -- > Dr Alexander Supalov > Intel GmbH > Hermuelheimer Strasse 8a > 50321 Bruehl, Germany > Phone: +49 2232 209034 > Mobile: +49 173 511 8735 > Fax: +49 2232 209029 > > > -----Original Message----- > From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org] = On > Behalf Of Erez Haba > Sent: Wednesday, January 09, 2008 8:12 PM > To: mpi-21@mpi-forum.org > Subject: RE: [mpi-21] Proposal EH2: add const keyword to the C bindin= gs > > Alexander, you really don't want to implement the sender side tricks = as > described below. This immediately breaks any program that async sends= > the same buffer to two or more ranks, e.g., > > MPI_Isend(buf.. rank=3D3.); > MPI_Isend(buf.. rank=3D4.); > > Although this is not compliant with the standard, you can be sure tha= t > many applications and commercial applications are using this pattern.= > > Thanks, > .Erez > > -----Original Message----- > From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org] = On > Behalf Of Supalov, Alexander > Sent: Wednesday, January 09, 2008 4:14 AM > To: mpi-21@mpi-forum.org > Subject: RE: [mpi-21] Proposal EH2: add const keyword to the C bindin= gs > > Hi, > > Let's not forget that gcc is not the only C compiler to consider. > There's Intel, Microsoft, PGI, and some other popular compilers. > > My primary problem with the const keyword is that it (seems to) prohi= bit > sender side tricks like the in-place little- to big-endian conversion= > before sending and back afterwards. > > And let's not forget that most Fortran bindings are built on top of C= > ones. What will happen to the copy-in/out done by some Fortran 90+ > compilers if the underlying C binding uses const declaration for the > inner C call, I don't know. > > Best regards. > > Alexander > > -- > Dr Alexander Supalov > Intel GmbH > Hermuelheimer Strasse 8a > 50321 Bruehl, Germany > Phone: +49 2232 209034 > Mobile: +49 173 511 8735 > Fax: +49 2232 209029 > > > -----Original Message----- > From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org] = On > Behalf Of Dries Kimpe > Sent: Wednesday, January 09, 2008 11:28 AM > To: mpi-21@mpi-forum.org > Subject: Re: [mpi-21] Proposal EH2: add const keyword to the C bindin= gs > > * Richard Treumann [2008-01-08 15:40:20]:> > > The MPI rules presumably would allow the block of code to access > buf[0] and > > buf[2] before and after the call to MPI_Send and the MPI_Send will = not > > change buf[0] or buf[2]. The practical effect of labeling buf[] co= nst > will > > be correct as long as the MPI rules are followed and there is no co= de > in > > the vicinity that references buf[1]. > > I'm a little bit worried about the fact that const buf[] (in C) spans= > the > whole array, while the MPI_Datatype only includes elements 0 and 2; (= And > even if char[] gets degraded to char * and only affects the first arr= ay > element, the problem remains when switching D1 and D2 in the example > below). > > This difference could(or could not) lead to problems, even if the use= r > follows the MPI rules. (Assuming Richard's interpretation of the con= st > keyword is 100% accurate) > > Given 2 mpi datatypes: > D1: byte, 1byte-hole byte > D2: 1byte-hole, byte > > Using these definitions, we only pass buf (and not buf[1]) to the MPI= > send > and receive functions > > so: > > char buf[3]; > > 1: some_nonconst_irecv_func (buf, D1); > 2: read buf[1] > 3: some_const_send_func (buf, D2); > 4: modify buf[1] > 5: some_const_send_func (buf, D2); > 6: some_complete_func (); > 7: read buf[0], buf[2] > > Even though the user doesn't access buf[0] or buf[2] between the rece= ive > start and the receive complete, the compiler might. > > Assume an architecture that doesn't do non-aligned memory accesses, > or one where byte access (write/read) is much slower than native-word= > accesses. > > In that case, the read on line 2 might actually read buf[0] and buf[1= ] > into a register. In the same fashion, the modify on line 4 might writ= e > buf[0] and buf[1] (maybe even using the buf[0] value stored in a > register) > > Even more so, since buf was only passed to const functions between li= ne > 2 > and 7, the compiler might assume on line 7 that it still has the corr= ect > value of buf[0] in a register. > > In that case, even though the user program conforms to the MPI rules,= > the resulting binary would be flawed. > > That being said, I've done some tests and couldn't trick gcc into > this behaviour. Maybe because gcc thinks it isn't slower to do byte > access on the 32 bit x86 machine (which doesn't have a problem with > unaligned memory access) i'm using right now, > --or-- > maybe because it has a rule that prohibits this behaviour. > > If it is the first, then we might have a problem om some architecture= s; > If it is the second, than the const shouldn't pose a problem. > > As for me, I'm thinking that this is what the restrict/and or volatil= e > keyword is for (and that the const definition given in the previous m= ail > is too severe). Unless I'm wrong, restrict indicates that the data is= n't > aliased somewhere else and that the compiler can safely assume the da= ta > won't be modified by a function call that doesn't have some (non-cons= t) > reference to the data as a parameter. Except when using restrict, usi= ng > a > const parameter doesn't buy you anything at all except some ease of m= ind > that a well-behaved compiler will inform you when you're trying to > modify > the const parameter within the function body. > > Besides, the fact that a function call is made using a const or nonco= nst > parameter is a non-issue all together if one allows > > ``The value of buf[1] can change asynchronously any time between > MPI_Irecv and MPI_Wait without any need for a function call.'' > > Greetings, > Dries > > PS. I've tested using the following program: > > int nonconstfunc (void * b); > int constfunc (const void * b); > > char b[4]; > > int main (int argc, char ** args) > { > b[0]=3D1; b[1]=3D2; b[2]=3D3; b[3]=3D4; > nonconstfunc(b); > b[0]+=3D1; /* cause read of B0 */ > constfunc(b); > b[0]=3Db[0]+b[2]; /* cause read of B2 */ > b[0]*=3D2; > constfunc(b); > } > > which compiles into (gcc4, -O3, -march=3Dprescott): > movl $b, (%esp) > call nonconstfunc > addb $1, b > movl $b, (%esp) > call constfunc > movzbl b+2, %eax > addb b, %al > movsbl %al,%eax > addl %eax, %eax > movb %al, b > movl $b, (%esp) > call constfunc > > ---------------------------------------------------------------------= > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for= > the sole use of the intended recipient(s). Any review or distribution= > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > ---------------------------------------------------------------------= > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for= > the sole use of the intended recipient(s). Any review or distribution= > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. >= --0__=0ABBF958DFE3719C8f9e8a93df938690918c0ABBF958DFE3719C Content-type: text/html; charset=US-ASCII Content-Disposition: inline Content-transfer-encoding: quoted-printable

An obvious question to me at least is: "Who is the user?"<= br>
The person who must write an MPI application and get it right it seems = to be the "user" Erez is thinking of.

The organization that buys and operates a parallel machine to solve pro= blems is also a "user".

The implementor of an MPI offering serves both kinds of user. Preservin= g specific freedoms for the implementor is not about making the impleme= ntor's job easy.

Let's pretend an MPI implementator who can assume the user will obey th= e standard could make MPI_Isend 5 times as fast as one that must prote= ct some user who ignores the standard. In this case, the second class o= f "user" would probably say. "Let the MPI programmer lea= rn the rules. I want the machine performance."

Now let's pretend everyone agrees the performance advantage of being ab= le to assume the user will always honor the rule will never exceed 0.00= 001% on any machine, now or in the future. In this case both types of= user and the implementor will all agree that the rule should be remove= d and the application writer relieved of the need to remember an "= unnatural" rule.

I said *pretend* in both cases because I know the real situation= is far less clear than the absurd performance numbers I used.

My point is that any proposed simplification of the standard will not s= imply be a trade off between

1) making life as easy as possible for people who write MPI libraries <= br> or
2) or making it as easy as possible for people who write MPI applicatio= ns.

I think it is likely the forum will conclude that the send buffer acces= s rules should be relaxed but there should be more to the discussion th= an "Do you care about users or don't you?"

Dick


Dick Treumann - MPI Team/TCEM
IBM Systems & Technology Group
Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
Tele (845) 433-7846 Fax (845) 433-8363


owner-mpi-21@mpi-forum.org wrote on 01/09/2008 02:58:21 PM:

> This issue is related to the "send buffer access". I say= "wait,
> let's make it right for the users by restricting the implementers&= quot;.
>
> The trick you described is legal with the current standard, but is=
> so surprisingly wrong for many of the users.
>
>
> -----Original Message-----
> From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org]
> On Behalf Of Supalov, Alexander
> Sent: Wednesday, January 09, 2008 11:43 AM
> To: mpi-21@mpi-forum.org
> Subject: RE: [mpi-21] Proposal EH2: add const keyword to the C bin= dings
>
> Wait, we're talking about standard here. The program you showed is= plain
> incorrect. The trick I described is legal. All according to the > standard. Let's keep our bearings straight: we don't want to make = broken
> programs work by reducing the freedom of implementors.
>
> --
> Dr Alexander Supalov
> Intel GmbH
> Hermuelheimer Strasse 8a
> 50321 Bruehl, Germany
> Phone:          +49 2232 209034
> Mobile:          +49 173 511 8735
> Fax:              +49 2232 2090= 29
>
>
> -----Original Message-----
> From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org] On
> Behalf Of Erez Haba
> Sent: Wednesday, January 09, 2008 8:12 PM
> To: mpi-21@mpi-forum.org
> Subject: RE: [mpi-21] Proposal EH2: add const keyword to the C bin= dings
>
> Alexander, you really don't want to implement the sender side tric= ks as
> described below. This immediately breaks any program that async se= nds
> the same buffer to two or more ranks, e.g.,
>
>         MPI_Isend(buf.. rank=3D3.);
>         MPI_Isend(buf.. rank=3D4.);
>
> Although this is not compliant with the standard, you can be sure = that
> many applications and commercial applications are using this patte= rn.
>
> Thanks,
> .Erez
>
> -----Original Message-----
> From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org] On
> Behalf Of Supalov, Alexander
> Sent: Wednesday, January 09, 2008 4:14 AM
> To: mpi-21@mpi-forum.org
> Subject: RE: [mpi-21] Proposal EH2: add const keyword to the C bin= dings
>
> Hi,
>
> Let's not forget that gcc is not the only C compiler to consider.<= br> > There's Intel, Microsoft, PGI, and some other popular compilers. >
> My primary problem with the const keyword is that it (seems to) pr= ohibit
> sender side tricks like the in-place little- to big-endian convers= ion
> before sending and back afterwards.
>
> And let's not forget that most Fortran bindings are built on top o= f C
> ones. What will happen to the copy-in/out done by some Fortran 90+=
> compilers if the underlying C binding uses const declaration for t= he
> inner C call, I don't know.
>
> Best regards.
>
> Alexander
>
> --
> Dr Alexander Supalov
> Intel GmbH
> Hermuelheimer Strasse 8a
> 50321 Bruehl, Germany
> Phone:          +49 2232 209034
> Mobile:          +49 173 511 8735
> Fax:              +49 2232 2090= 29
>
>
> -----Original Message-----
> From: owner-mpi-21@mpi-forum.org [mailto:owner-mpi-21@mpi-forum.org] On
> Behalf Of Dries Kimpe
> Sent: Wednesday, January 09, 2008 11:28 AM
> To: mpi-21@mpi-forum.org
> Subject: Re: [mpi-21] Proposal EH2: add const keyword to the C bin= dings
>
> * Richard Treumann <treumann@XXXXXXXXXXXXXX [2008-01-08 15:40:2= 0]:
>
> > The MPI rules presumably would allow the block of code to acc= ess
> buf[0] and
> > buf[2] before and after the call to MPI_Send and the MPI_Send= will not
> > change buf[0] or buf[2].  The practical effect of labeli= ng buf[] const
> will
> > be correct as long as the MPI rules are followed and there is= no code
> in
> > the vicinity that references buf[1].
>
> I'm a little bit worried about the fact that const buf[] (in C) sp= ans
> the
> whole array, while the MPI_Datatype only includes elements 0 and 2= ; (And
> even if char[] gets degraded to char * and only affects the first = array
> element, the problem remains when switching D1 and D2 in the examp= le
> below).
>
> This difference could(or could not) lead to problems, even if the = user
> follows the MPI rules. (Assuming Richard's  interpretation of= the const
> keyword is 100% accurate)
>
> Given 2 mpi datatypes:
>   D1:  byte, 1byte-hole byte
>   D2: 1byte-hole, byte
>
> Using these definitions, we only pass buf (and not buf[1]) to the = MPI
> send
> and receive functions
>
> so:
>
> char buf[3];
>
> 1: some_nonconst_irecv_func (buf, D1);
> 2: read buf[1]
> 3: some_const_send_func (buf, D2);
> 4: modify buf[1]
> 5: some_const_send_func (buf, D2);
> 6: some_complete_func ();
> 7: read buf[0], buf[2]
>
> Even though the user doesn't access buf[0] or buf[2] between the r= eceive
> start and the receive complete, the compiler might.
>
> Assume an architecture that doesn't do non-aligned memory accesses= ,
> or one where byte access (write/read) is much slower than native-w= ord
> accesses.
>
> In that case, the read on line 2 might actually read buf[0] and bu= f[1]
> into a register. In the same fashion, the modify on line 4 might w= rite
> buf[0] and buf[1] (maybe even using the buf[0] value stored in a > register)
>
> Even more so, since buf was only passed to const functions between= line
> 2
> and 7, the compiler might assume on line 7 that it still has the c= orrect
> value of buf[0] in a register.
>
> In that case, even though the user program conforms to the MPI rul= es,
> the resulting binary would be flawed.
>
> That being said, I've done some tests and couldn't trick gcc into<= br> > this behaviour. Maybe because gcc thinks it isn't slower to do byt= e
> access on the 32 bit x86 machine (which doesn't have a problem wit= h
> unaligned memory access) i'm using right now,
> --or--
> maybe because it has a rule that prohibits this behaviour.
>
> If it is the first, then we might have a problem om some architect= ures;
> If it is the second, than the const shouldn't pose a problem.
>
> As for me, I'm thinking that this is what the restrict/and or vola= tile
> keyword is for (and that the const definition given in the previou= s mail
> is too severe). Unless I'm wrong, restrict indicates that the data= isn't
> aliased somewhere else and that the compiler can safely assume the= data
> won't be modified by a function call that doesn't have some (non-c= onst)
> reference to the data as a parameter. Except when using restrict, = using
> a
> const parameter doesn't buy you anything at all except some ease o= f mind
> that a well-behaved compiler will inform you when you're trying to=
> modify
> the const parameter within the function body.
>
> Besides, the fact that a function call is made using a const or no= nconst
> parameter is a non-issue all together if one allows
>
> ``The value of buf[1] can change asynchronously any time between > MPI_Irecv and MPI_Wait without any need for a function call.''
= >
>   Greetings,
>   Dries
>
> PS. I've tested using the following program:
>
> int nonconstfunc (void * b);
> int constfunc (const void * b);
>
> char  b[4];
>
> int main (int argc, char ** args)
> {
>    b[0]=3D1; b[1]=3D2; b[2]=3D3; b[3]=3D4;
>    nonconstfunc(b);
>    b[0]+=3D1;   /* cause read of B0 */
>    constfunc(b);
>    b[0]=3Db[0]+b[2];   /* cause read of B2 */
>    b[0]*=3D2;
>    constfunc(b);
> }
>
> which compiles into (gcc4, -O3, -march=3Dprescott):
>    movl    $b, (%esp)
>    call    nonconstfunc
>    addb    $1, b
>    movl    $b, (%esp)
>    call    constfunc
>    movzbl  b+2, %eax
>    addb    b, %al
>    movsbl  %al,%eax
>    addl    %eax, %eax
>    movb    %al, b
>    movl    $b, (%esp)
>    call    constfunc
>
> ------------------------------------------------------------------= ---
> Intel GmbH
> Dornacher Strasse 1
> 85622 Feldkirchen/Muenchen Germany
> Sitz der Gesellschaft: Feldkirchen bei Muenchen
> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwadere= r
> Registergericht: Muenchen HRB 47456 Ust.-IdNr.
> VAT Registration No.: DE129385895
> Citibank Frankfurt (BLZ 502 109 00) 600119052
>
> This e-mail and any attachments may contain confidential material = for
> the sole use of the intended recipient(s). Any review or distribut= ion
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
> ------------------------------------------------------------------= ---
> Intel GmbH
> Dornacher Strasse 1
> 85622 Feldkirchen/Muenchen Germany
> Sitz der Gesellschaft: Feldkirchen bei Muenchen
> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwadere= r
> Registergericht: Muenchen HRB 47456 Ust.-IdNr.
> VAT Registration No.: DE129385895
> Citibank Frankfurt (BLZ 502 109 00) 600119052
>
> This e-mail and any attachments may contain confidential material = for
> the sole use of the intended recipient(s). Any review or distribut= ion
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
= --0__=0ABBF958DFE3719C8f9e8a93df938690918c0ABBF958DFE3719C--