Any simple way to do for loop pipe , such as ` ls | grep ".*.7z" | 7z -e {}`

I am curious that is there an function/command to cope for loop pipe ?

It is easy to occur below demand

1. you run a command return multiple line
2. you need process each line
3. pass each line to another function/command

For example:

you need find some path match a parttern , move each to another place :

# 1.
find path_A -name '*AAA*' -exec mv -t path_B {} +
# 2.
find path_A -name "*AAA*" -print0 | xargs -0 -I {} mv {} path_B 

People would be confused,

1. why need mv -t (I mean, through a [for loop pipe] you can use mv {} some_path ，just like what you wrote without pipe )
2. why exec or | xargs is write in that way , xargs need much more options, but they are actually no need in many computer language 。
3. why I can't use ls ... mv -t path_B {} + , or ls | xargs -0 -I {} mv {} path_B , in other words , why it doesn't work after I change to another command ?
4. Why I failed when change mv to 7z : find path_A -name '*AAA*' -exec 7z -t {} + ?

NOTE: Above are not question, just examples!!

---

I asked this question for a common and simple way to cope for loop pipe . Is there a simple and common way to do:

# 0
command with option (usually you use a command like this)
# 1
command with option [for loop pipe] command with option | command with option
# 2
command with option [for loop pipe] command with option | command with option [for loop pipe] command with option
# 3
....

So you don't need change command with option when you use [for loop pipe] . This would be much convenient for daily use .

Update

I put all the example just want to say normal way is confusing when different comand has it own style to cope with multiline input, it would need exec or xargs or some else options . if we have a for pipe command which can connect two command with option . It would be much easy to use

I mean, if you are using an advanced language. For example python,

def pipe(x, func, *func_options): return func(x, *func_options)
def forloop_pipe(x, func, *func_options): return [func(i, *func_options) for i in x.split('\n')]
def ls(*options) pass
def mv(*options) pass
a = ls('*.txt')
b = forloop_pipe(a, mv, '/home')

This is a very simple example with a lot of flaws, but it explains what I'm trying to get in bash command .

7

3 Answers

I'm not sure this is a straight answer to the question (there are much more than one questions), but I think you are asking for something like:

• for loop with command substitution or
• while loop with process substitution.

for loop with $(command substitution)

Command substitution allows the output of a command to replace the command itself. We can combine this functionality with the for loop in this way:

for item in "$(find . -type f)"
do echo "$item" | tee -a ./"file-list.txt"
done

while loop with <(process substitution)

Process substitution allows a process’s input or output to be referred to using a filename. We can combine this functionality with the while loop, by the help of the built in read, in this way:

while IFS= read -r item
do echo "$item" | tee -a ./"file-list.txt"
done < <(find . -type f)

Within the final part < <(find . -type f), the first < means the stdin of the while loop and <(find . -type f) will be treated as a file. More about the construction IFS= read -r line you can read in this great post of Stéphane Chazelas.

You can achieve the same as the above by using pipe instead of process substitution (but this option is not preferred, because it could cause errors):

find . -type f | while IFS= read -r file
do echo "$file" | tee -a ./"file-list.txt"
done

Pipe the output of a loop to another command

Further we can pipe or redirect the output of the loop to another command or function:

while IFS= read -r -d '' item; do # Compose the name of the new file DIR="$(dirname "${item}")" FILE_NAME="$(basename "${item}")" NEW_FILE_NAME="new-${FILE_NAME}" # move the file and suppress the potential output # of the 'mv' command by redirecting it to a log file mv "${item}" "${DIR}/${NEW_FILE_NAME}" >>/tmp/mv-loop.log 2>&1 # output the name of the new file in order to be processed # by the next (piped) command printf '%b' "${DIR}/${NEW_FILE_NAME}"'\0'
done < <(find . -type f -print0) | xargs -0 -I{} 7z a -t7z "the_archive_name.7z" {}

Note within the above example a null delimiters is used everywhere:

• find . -print0,
• IFS= read -r -d '' item - reference,
• printf '%b' a'\0'b - reference,
• xargs -0 (in this case we do not actually need -I{} and {} within the xargs command).

This example will move (rename) all files recursively, but within the archive the directory structure wont appear.

1

Is there a simple and common way to do

Answer: No, every command is different.

---

Regarding your questions:

1. You don't need -t, but then mv will run for each file, which will be a lot slower. You need to end your -exec with \; instead of +.

2. A pipe has nothing to do with a for loop, a pipe sends stdout from left side of the pipe to stdin on the right side. xargs reads from stdin (or from a file with -a option) and runs a command with arguments taken from the input.

3. ls has no -exec option. ls will work with xargs, but not with xargs -0 because ls has no option for NULL delimited output. That is also why you should not use ls with pipes at all (file names are allowed to have newlines).

4. For mv, -t is target directory, while for 7z it is archive type, why do you think it should work?

For me it seems you're using commands you do not really understand. This won't work for any programming or scripting language.

---

Btw: You should use xargs with -r option to avoid issues with empty input.

1

Maybe your answer is xargs -n 1

See this example: 1 only line (loop generated for gzip):

ls | grep -i '.csv' | xargs -n 1 gzip

or

ls | grep -i '.gz' | xargs -n 1 gunzip

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy