using cat to get a list to for in sh or in ksh

Discussion in 'Programmer Misc' started by Robert Peirce, Feb 6, 2008.

  1. In sh and ksh, I have used

    for i in `cat file`
    do
    do-something-with-$i
    done

    on other systems with no problem. In OS X, this sometimes hangs, but
    sometimes it doesn't, which is very confusing. Pr also hangs, but for
    some strange reason, head works in the same script!

    Does anybody have any idea what I am doing wrong? This should always
    work fine, but sometimes it doesn't.

    --
    Robert B. Peirce, Venetia, PA 724-941-6883
    bob AT peirce-family.com [Mac]
    rbp AT cooksonpeirce.com [Office]
     
    Robert Peirce, Feb 6, 2008
    #1
    1. Advertisements

  2. In article <>,
    Robert Peirce <> wrote:

    > In sh and ksh, I have used
    >
    > for i in `cat file`
    > do
    > do-something-with-$i
    > done
    >
    > on other systems with no problem. In OS X, this sometimes hangs, but
    > sometimes it doesn't, which is very confusing. Pr also hangs, but for
    > some strange reason, head works in the same script!
    >
    > Does anybody have any idea what I am doing wrong? This should always
    > work fine, but sometimes it doesn't.


    Okay. I have more info. There seems to be some sort of buffer
    limitation in for. Perhaps somebody can tell me what it is. The file,
    in the case that hangs, is 1,635,888 lines and 24,538,320 characters.
    Other platforms seem to be able to handle this. OS X is hanging.

    --
    Robert B. Peirce, Venetia, PA 724-941-6883
    bob AT peirce-family.com [Mac]
    rbp AT cooksonpeirce.com [Office]
     
    Robert Peirce, Feb 6, 2008
    #2
    1. Advertisements

  3. In article <>,
    Robert Peirce <> wrote:

    > In article <>,
    > Robert Peirce <> wrote:
    >
    > > In sh and ksh, I have used
    > >
    > > for i in `cat file`
    > > do
    > > do-something-with-$i
    > > done
    > >
    > > on other systems with no problem. In OS X, this sometimes hangs, but
    > > sometimes it doesn't, which is very confusing. Pr also hangs, but for
    > > some strange reason, head works in the same script!
    > >
    > > Does anybody have any idea what I am doing wrong? This should always
    > > work fine, but sometimes it doesn't.

    >
    > Okay. I have more info. There seems to be some sort of buffer
    > limitation in for. Perhaps somebody can tell me what it is. The file,
    > in the case that hangs, is 1,635,888 lines and 24,538,320 characters.
    > Other platforms seem to be able to handle this. OS X is hanging.


    This is embarrassing (for OS X), but I ran this script using ksh in Uwin
    on a PC and it worked:

    for i in $(cat s.p); do print $i; done | head

    but it doesn't work in ksh on OS X.

    Cat works fine by itself, so the limitation is in the for;do;done loop.

    --
    Robert B. Peirce, Venetia, PA 724-941-6883
    bob AT peirce-family.com [Mac]
    rbp AT cooksonpeirce.com [Office]
     
    Robert Peirce, Feb 6, 2008
    #3
  4. Robert Peirce <> writes:

    >In sh and ksh, I have used


    > for i in `cat file`
    > do
    > do-something-with-$i
    > done


    >on other systems with no problem. In OS X, this sometimes hangs, but
    >sometimes it doesn't, which is very confusing. Pr also hangs, but for
    >some strange reason, head works in the same script!


    Reading the whole file into one line exceeds the shell's line length
    limit.

    It works on other systems because they have longer line-length limits.
    It's still a bad idea.

    head(1) works because it's not reading the whole file.

    The ksh(1) input mechanism is the read builtin, not cat(1). RTFM
    ksh(1).

    --
    dhs
     
    David Spencer, Feb 6, 2008
    #4
  5. Robert Peirce

    Bob Harris Guest

    In article <>,
    Robert Peirce <> wrote:

    > In sh and ksh, I have used
    >
    > for i in `cat file`
    > do
    > do-something-with-$i
    > done
    >
    > on other systems with no problem. In OS X, this sometimes hangs, but
    > sometimes it doesn't, which is very confusing. Pr also hangs, but for
    > some strange reason, head works in the same script!
    >
    > Does anybody have any idea what I am doing wrong? This should always
    > work fine, but sometimes it doesn't.


    May I suggest

    while read line
    do
    do-something-with-$line
    done <file

    This will do something similar, assuming there is one item per
    line.

    if you have multiple fixed items per line you can do

    while read item1 item2 item3
    do
    do-something-with-$item1
    do-something-with-$item2
    do-something-with-$item3
    done <file

    Or if you do not know how may items are on each line you can try

    while read line
    do
    set -- $line
    for item
    do
    do-something-with-$item
    done
    done <file

    Bob Harris
     
    Bob Harris, Feb 7, 2008
    #5
  6. Bob Harris <-Harris.us> writes:

    > while read line
    > do
    > set -- $line
    > for item
    > do
    > do-something-with-$item
    > done
    > done <file


    tr -s '[:space:]' '\n' <file |
    while read x
    do command $x
    done

    --
    dhs
     
    David Spencer, Feb 7, 2008
    #6
  7. In article <fodebg$c4t$>,
    David Spencer <> wrote:

    > Robert Peirce <> writes:
    >
    > >In sh and ksh, I have used

    >
    > > for i in `cat file`
    > > do
    > > do-something-with-$i
    > > done

    >
    > >on other systems with no problem. In OS X, this sometimes hangs, but
    > >sometimes it doesn't, which is very confusing. Pr also hangs, but for
    > >some strange reason, head works in the same script!

    >
    > Reading the whole file into one line exceeds the shell's line length
    > limit.
    >
    > It works on other systems because they have longer line-length limits.
    > It's still a bad idea.
    >
    > head(1) works because it's not reading the whole file.
    >
    > The ksh(1) input mechanism is the read builtin, not cat(1). RTFM
    > ksh(1).


    I am familiar with read and I have used it in more recent scripts. This
    is an old one I thought would work and it didn't. As a stopgap, I used
    split to break it into smaller files that I could read with cat until I
    get around to using read.

    --
    Robert B. Peirce, Venetia, PA 724-941-6883
    bob AT peirce-family.com [Mac]
    rbp AT cooksonpeirce.com [Office]
     
    Robert Peirce, Feb 7, 2008
    #7
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jeffrey Rolland

    Installing ksh from AT&T

    Jeffrey Rolland, Mar 11, 2006, in forum: Apps
    Replies:
    5
    Views:
    157
    Warren Oates
    Mar 12, 2006
  2. Replies:
    11
    Views:
    217
    Paul Nevai
    May 30, 2007
  3. Robert Peirce

    Panther and ksh miss-match

    Robert Peirce, Jan 31, 2004, in forum: Misc
    Replies:
    2
    Views:
    127
    Robert Peirce
    Feb 2, 2004
  4. Robert Peirce

    ksh scripting problem

    Robert Peirce, Feb 8, 2008, in forum: Programmer Misc
    Replies:
    3
    Views:
    345
    Robert Peirce
    Feb 8, 2008
  5. Robert Peirce

    Odd ksh problem

    Robert Peirce, Feb 29, 2008, in forum: Programmer Misc
    Replies:
    5
    Views:
    365
    Robert Peirce
    Mar 2, 2008
  6. Replies:
    5
    Views:
    165
  7. Kurt R. Todoroff
    Replies:
    1
    Views:
    416
    David Empson
    Sep 16, 2009
  8. Martin S Taylor

    Top Cat [about as far OT as you can get]

    Martin S Taylor, Aug 17, 2005, in forum: UK Macs
    Replies:
    29
    Views:
    261
    Jon B
    Aug 24, 2005
Loading...