using cat to get a list to for in sh or in ksh

Discussion in 'Programmer Misc' started by Robert Peirce, Feb 6, 2008.

  1. In sh and ksh, I have used

    for i in `cat file`
    do
    do-something-with-$i
    done

    on other systems with no problem. In OS X, this sometimes hangs, but
    sometimes it doesn't, which is very confusing. Pr also hangs, but for
    some strange reason, head works in the same script!

    Does anybody have any idea what I am doing wrong? This should always
    work fine, but sometimes it doesn't.
     
    Robert Peirce, Feb 6, 2008
    #1
    1. Advertisements

  2. Okay. I have more info. There seems to be some sort of buffer
    limitation in for. Perhaps somebody can tell me what it is. The file,
    in the case that hangs, is 1,635,888 lines and 24,538,320 characters.
    Other platforms seem to be able to handle this. OS X is hanging.
     
    Robert Peirce, Feb 6, 2008
    #2
    1. Advertisements

  3. This is embarrassing (for OS X), but I ran this script using ksh in Uwin
    on a PC and it worked:

    for i in $(cat s.p); do print $i; done | head

    but it doesn't work in ksh on OS X.

    Cat works fine by itself, so the limitation is in the for;do;done loop.
     
    Robert Peirce, Feb 6, 2008
    #3
  4. Reading the whole file into one line exceeds the shell's line length
    limit.

    It works on other systems because they have longer line-length limits.
    It's still a bad idea.

    head(1) works because it's not reading the whole file.

    The ksh(1) input mechanism is the read builtin, not cat(1). RTFM
    ksh(1).
     
    David Spencer, Feb 6, 2008
    #4
  5. Robert Peirce

    Bob Harris Guest

    May I suggest

    while read line
    do
    do-something-with-$line
    done <file

    This will do something similar, assuming there is one item per
    line.

    if you have multiple fixed items per line you can do

    while read item1 item2 item3
    do
    do-something-with-$item1
    do-something-with-$item2
    do-something-with-$item3
    done <file

    Or if you do not know how may items are on each line you can try

    while read line
    do
    set -- $line
    for item
    do
    do-something-with-$item
    done
    done <file

    Bob Harris
     
    Bob Harris, Feb 7, 2008
    #5
  6. tr -s '[:space:]' '\n' <file |
    while read x
    do command $x
    done
     
    David Spencer, Feb 7, 2008
    #6
  7. I am familiar with read and I have used it in more recent scripts. This
    is an old one I thought would work and it didn't. As a stopgap, I used
    split to break it into smaller files that I could read with cat until I
    get around to using read.
     
    Robert Peirce, Feb 7, 2008
    #7
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.