Bash - pair each line of file2019 Community Moderator ElectionCommand line tool to “cat” pairwise expansion of all rows in a fileCost-efficiently pair each line of a file with all othersHow to rewrite multiline path into one-line relative pathMerge two files line by line with the delimiter triple pipe symbol “|||”How do I remove all but the file name (with no extension) from a full file path?Insert string or line after last instance of a specific search variable, in a loopcopy every line from a text file that contains a number greater than 5000How to insert a line from file A above the FIRST LINE in file Bextract lines from a file based on sequential pair of patterns, and output to separate filesscript to parse file for two consecutive lines of unequal lengthHow to add a path before the first character of each line in a file.txt and save the same file with those edits?How can I add 10 lines from a file (file2) to another one after 2 lines (file1)?
Using an older 200A breaker panel on a 60A feeder circuit from house?
Taking the numerator and the denominator
Offset in split text content
How to test the sharpness of a knife?
Recursively move files within sub directories
categorizing a variable turns it from insignificant to significant
Rendered textures different to 3D View
Should a narrator ever describe things based on a character's view instead of facts?
I keep switching characters, how do I stop?
Why didn’t Eve recognize the little cockroach as a living organism?
Put the phone down / Put down the phone
Are hand made posters acceptable in Academia?
Why does the frost depth increase when the surface temperature warms up?
Checking @@ROWCOUNT failing
What is the period/term used describe Giuseppe Arcimboldo's style of painting?
What is this high flying aircraft over Pennsylvania?
1 John in Luther’s Bibel
Amorphous proper classes in MK
What is it called when someone votes for an option that's not their first choice?
Magnifying glass in hyperbolic space
How can I, as DM, avoid the Conga Line of Death occurring when implementing some form of flanking rule?
How would a solely written language work mechanically
Has the laser at Magurele, Romania reached a tenth of the Sun's power?
How do you say "Trust your struggle." in French?
Bash - pair each line of file
2019 Community Moderator ElectionCommand line tool to “cat” pairwise expansion of all rows in a fileCost-efficiently pair each line of a file with all othersHow to rewrite multiline path into one-line relative pathMerge two files line by line with the delimiter triple pipe symbol “|||”How do I remove all but the file name (with no extension) from a full file path?Insert string or line after last instance of a specific search variable, in a loopcopy every line from a text file that contains a number greater than 5000How to insert a line from file A above the FIRST LINE in file Bextract lines from a file based on sequential pair of patterns, and output to separate filesscript to parse file for two consecutive lines of unequal lengthHow to add a path before the first character of each line in a file.txt and save the same file with those edits?How can I add 10 lines from a file (file2) to another one after 2 lines (file1)?
This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B
is equal to a B A
pair for my purposes, so only one of these combinations should be produced.
Example
files.dat
reads like this in a shorthand notation, each letter is a file path (absolute or relative)
a
b
c
d
e
Then my result should look something like this:
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.
shell-script text-processing
New contributor
add a comment |
This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B
is equal to a B A
pair for my purposes, so only one of these combinations should be produced.
Example
files.dat
reads like this in a shorthand notation, each letter is a file path (absolute or relative)
a
b
c
d
e
Then my result should look something like this:
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.
shell-script text-processing
New contributor
Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.
– Jeff Schaller
2 days ago
@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry
– Enno
2 days ago
This is almost becoming a Code Golf :P
– Richard de Wit
yesterday
1
As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).
– Davidmh
yesterday
add a comment |
This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B
is equal to a B A
pair for my purposes, so only one of these combinations should be produced.
Example
files.dat
reads like this in a shorthand notation, each letter is a file path (absolute or relative)
a
b
c
d
e
Then my result should look something like this:
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.
shell-script text-processing
New contributor
This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B
is equal to a B A
pair for my purposes, so only one of these combinations should be produced.
Example
files.dat
reads like this in a shorthand notation, each letter is a file path (absolute or relative)
a
b
c
d
e
Then my result should look something like this:
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.
shell-script text-processing
shell-script text-processing
New contributor
New contributor
edited 2 days ago
Jeff Schaller
43.5k1161140
43.5k1161140
New contributor
asked 2 days ago
EnnoEnno
1483
1483
New contributor
New contributor
Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.
– Jeff Schaller
2 days ago
@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry
– Enno
2 days ago
This is almost becoming a Code Golf :P
– Richard de Wit
yesterday
1
As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).
– Davidmh
yesterday
add a comment |
Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.
– Jeff Schaller
2 days ago
@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry
– Enno
2 days ago
This is almost becoming a Code Golf :P
– Richard de Wit
yesterday
1
As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).
– Davidmh
yesterday
Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.
– Jeff Schaller
2 days ago
Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.
– Jeff Schaller
2 days ago
@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry
– Enno
2 days ago
@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry
– Enno
2 days ago
This is almost becoming a Code Golf :P
– Richard de Wit
yesterday
This is almost becoming a Code Golf :P
– Richard de Wit
yesterday
1
1
As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).
– Davidmh
yesterday
As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).
– Davidmh
yesterday
add a comment |
6 Answers
6
active
oldest
votes
Use this command:
awk ' name[$1]++
END PROCINFO["sorted_in"] = "@ind_str_asc"
for (v1 in name) for (v2 in name) if (v1 < v2) print v1, v2
' files.dat
PROCINFO
may be a gawk
extension.
If your awk
doesn’t support it,
just leave out the PROCINFO["sorted_in"] = "@ind_str_asc"
line
and pipe the output into sort
(if you want the output sorted).
(This does not require the input to be sorted.)
add a comment |
$ join -j 2 -o 1.1,2.1 file file | awk '!seen[$1,$2]++ && !seen[$2,$1]++'
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
This assumes that no line in the input file contains any whitespace. It also assumes that the file is sorted.
The join
command creates the full cross product of the lines in the file. It does this by joining the file with itself on a non-existing field. The non-standard -j 2
may be replaced by -1 2 -2 2
(but not by -j2
unless you use GNU join
).
The awk
command reads the result of this and only outputs results that are pairs that has not yet been seen.
What do you mean by "the file is sorted"? Sorted by which criteria?
– Enno
2 days ago
@Enno Sorted the waysort -b
would sort it.join
require sorted input files.
– Kusalananda
2 days ago
add a comment |
If you have ruby
installed:
$ ruby -0777 -F'n' -lane '$F.combination(2) puts c.join(" ")' ip.txt
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
-0777
slurp entire file (should be okay as it is mentioned in OP that file size is small)-F'n'
split based on newline, so each line will be an element in$F
array$F.combination(2)
generate combinations2
elements at a timeputs c.join(" ")
print as required- if input file can contain duplicates, use
$F.uniq.combination(2)
for 3 elements at a time:
$ ruby -0777 -F'n' -lane '$F.combination(3) puts c.join(" ")' ip.txt
a b c
a b d
a b e
a c d
a c e
a d e
b c d
b c e
b d e
c d e
With perl
(not generic)
$ perl -0777 -F'n' -lane 'for $i (0..$#F)
for $j ($i+1..$#F)
print "$F[$i] $F[$j]n" ' ip.txt
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
With awk
$ awk ' a[NR]=$0
END for(i=1;i<=NR;i++)
for(j=i+1;j<=NR;j++)
print a[i], a[j] ' ip.txt
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
add a comment |
A python
solution.
The input file is fed to itertools.combinations
from the standard library, which generates 2-length tuples that are formatted and printed to standard output.
python3 -c 'from itertools import combinations
with open("file") as f:
lines = (line.rstrip() for line in f)
lines = (" ".format(x, y) for x, y in combinations(lines, 2))
print(*lines, sep="n")
'
add a comment |
Here's one in pure shell.
test $# -gt 1 || exit
a=$1
shift
for f in "$@"
do
echo $a $f
done
exec /bin/sh $0 "$@"
Example:
~ (137) $ sh test.sh $(cat file.dat)
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
~ (138) $
New contributor
1
Command substitution strips trailing newlines, so you're better off with something like<file.dat xargs test.sh
thantest.sh $(cat file.dat)
– iruvar
2 days ago
add a comment |
Using Perl
we can do it as shown:
$ perl -lne '
push @A, $_} exit
a=$1
shift
for f in "$@"
do
echo $a $f
done
exec /bin/sh $0 "$@"
Example:
~ (137) $ sh test.sh $(cat file.dat)
a b
a c
a d
a e
b c
b d
b e
c d
c e
d e
~ (138) $
add a comment
while ( @A )
my $e = shift @A;
print "$e $_" for @A;
' input.txt
answered 2 days ago
Rakesh SharmaRakesh Sharma
342115
342115
add a comment |
add a comment |
Enno is a new contributor. Be nice, and check out our Code of Conduct.
Enno is a new contributor. Be nice, and check out our Code of Conduct.
Enno is a new contributor. Be nice, and check out our Code of Conduct.
Enno is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f506815%2fbash-pair-each-line-of-file%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.
– Jeff Schaller
2 days ago
@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry
– Enno
2 days ago
This is almost becoming a Code Golf :P
– Richard de Wit
yesterday
1
As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).
– Davidmh
yesterday