Find the identical rows in a matrix [duplicate]












6












$begingroup$



This question already has an answer here:




  • How to efficiently find positions of duplicates?

    8 answers




Suppose I have the following matrix:



M = 
{{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0,1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}};

TableForm[M, TableHeadings -> {{S1, S2, S3, S4, S5, S6, S7, S8}}]


matrix



In this case, it turns out that rows (S1, S8), (S2, S3, S4), (S5, S6, S7) have equal element values in identical column positions. I have a 1000 x 1000 matrix to examine and would appreciate any assistance in coding this problem.










share|improve this question











$endgroup$



marked as duplicate by Michael E2, m_goldberg, happy fish, bbgodfrey, C. E. May 1 at 13:07


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.














  • 3




    $begingroup$
    Try Values[PositionIndex[M]]
    $endgroup$
    – Coolwater
    Apr 25 at 21:39










  • $begingroup$
    @Coolwater If there is a unique row, your method will fail. At least one needs to delete if list has length 1
    $endgroup$
    – Okkes Dulgerci
    Apr 25 at 22:58










  • $begingroup$
    @Coolwater IMHO, the best answer is lacking so far. Please consider posting PositionIndex as possible solution.
    $endgroup$
    – Henrik Schumacher
    Apr 26 at 12:08
















6












$begingroup$



This question already has an answer here:




  • How to efficiently find positions of duplicates?

    8 answers




Suppose I have the following matrix:



M = 
{{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0,1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}};

TableForm[M, TableHeadings -> {{S1, S2, S3, S4, S5, S6, S7, S8}}]


matrix



In this case, it turns out that rows (S1, S8), (S2, S3, S4), (S5, S6, S7) have equal element values in identical column positions. I have a 1000 x 1000 matrix to examine and would appreciate any assistance in coding this problem.










share|improve this question











$endgroup$



marked as duplicate by Michael E2, m_goldberg, happy fish, bbgodfrey, C. E. May 1 at 13:07


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.














  • 3




    $begingroup$
    Try Values[PositionIndex[M]]
    $endgroup$
    – Coolwater
    Apr 25 at 21:39










  • $begingroup$
    @Coolwater If there is a unique row, your method will fail. At least one needs to delete if list has length 1
    $endgroup$
    – Okkes Dulgerci
    Apr 25 at 22:58










  • $begingroup$
    @Coolwater IMHO, the best answer is lacking so far. Please consider posting PositionIndex as possible solution.
    $endgroup$
    – Henrik Schumacher
    Apr 26 at 12:08














6












6








6


2



$begingroup$



This question already has an answer here:




  • How to efficiently find positions of duplicates?

    8 answers




Suppose I have the following matrix:



M = 
{{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0,1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}};

TableForm[M, TableHeadings -> {{S1, S2, S3, S4, S5, S6, S7, S8}}]


matrix



In this case, it turns out that rows (S1, S8), (S2, S3, S4), (S5, S6, S7) have equal element values in identical column positions. I have a 1000 x 1000 matrix to examine and would appreciate any assistance in coding this problem.










share|improve this question











$endgroup$





This question already has an answer here:




  • How to efficiently find positions of duplicates?

    8 answers




Suppose I have the following matrix:



M = 
{{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0,1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0},
{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}};

TableForm[M, TableHeadings -> {{S1, S2, S3, S4, S5, S6, S7, S8}}]


matrix



In this case, it turns out that rows (S1, S8), (S2, S3, S4), (S5, S6, S7) have equal element values in identical column positions. I have a 1000 x 1000 matrix to examine and would appreciate any assistance in coding this problem.





This question already has an answer here:




  • How to efficiently find positions of duplicates?

    8 answers








list-manipulation matrix






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Apr 25 at 20:03









m_goldberg

90.4k873203




90.4k873203










asked Apr 25 at 19:15









PRGPRG

1248




1248




marked as duplicate by Michael E2, m_goldberg, happy fish, bbgodfrey, C. E. May 1 at 13:07


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.









marked as duplicate by Michael E2, m_goldberg, happy fish, bbgodfrey, C. E. May 1 at 13:07


This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.










  • 3




    $begingroup$
    Try Values[PositionIndex[M]]
    $endgroup$
    – Coolwater
    Apr 25 at 21:39










  • $begingroup$
    @Coolwater If there is a unique row, your method will fail. At least one needs to delete if list has length 1
    $endgroup$
    – Okkes Dulgerci
    Apr 25 at 22:58










  • $begingroup$
    @Coolwater IMHO, the best answer is lacking so far. Please consider posting PositionIndex as possible solution.
    $endgroup$
    – Henrik Schumacher
    Apr 26 at 12:08














  • 3




    $begingroup$
    Try Values[PositionIndex[M]]
    $endgroup$
    – Coolwater
    Apr 25 at 21:39










  • $begingroup$
    @Coolwater If there is a unique row, your method will fail. At least one needs to delete if list has length 1
    $endgroup$
    – Okkes Dulgerci
    Apr 25 at 22:58










  • $begingroup$
    @Coolwater IMHO, the best answer is lacking so far. Please consider posting PositionIndex as possible solution.
    $endgroup$
    – Henrik Schumacher
    Apr 26 at 12:08








3




3




$begingroup$
Try Values[PositionIndex[M]]
$endgroup$
– Coolwater
Apr 25 at 21:39




$begingroup$
Try Values[PositionIndex[M]]
$endgroup$
– Coolwater
Apr 25 at 21:39












$begingroup$
@Coolwater If there is a unique row, your method will fail. At least one needs to delete if list has length 1
$endgroup$
– Okkes Dulgerci
Apr 25 at 22:58




$begingroup$
@Coolwater If there is a unique row, your method will fail. At least one needs to delete if list has length 1
$endgroup$
– Okkes Dulgerci
Apr 25 at 22:58












$begingroup$
@Coolwater IMHO, the best answer is lacking so far. Please consider posting PositionIndex as possible solution.
$endgroup$
– Henrik Schumacher
Apr 26 at 12:08




$begingroup$
@Coolwater IMHO, the best answer is lacking so far. Please consider posting PositionIndex as possible solution.
$endgroup$
– Henrik Schumacher
Apr 26 at 12:08










4 Answers
4






active

oldest

votes


















2












$begingroup$

I'd use GroupBy.



First the names of the rows: can be anything you like, for example



rownames = Array[ToExpression["S" <> ToString[#]] &, Length[M]]



{S1, S2, S3, S4, S5, S6, S7, S8}




Next the grouping:



groups = GroupBy[Thread[rownames -> M], Last -> First]



<|{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} -> {S1, S8},
{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0} -> {S2, S3, S4},
{0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0} -> {S5, S6, S7}|>




If all you need are the names:



Values[groups]



{{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







share|improve this answer









$endgroup$





















    8












    $begingroup$

    idx = DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]



    {{1, 8}, {2, 3, 4}, {5, 6, 7}}




    In order to obtain the labels of the rows, you may use the following:



    labels = {S1, S2, S3, S4, S5, S6, S7, S8};
    Map[labels[[#]] &, idx, {2}]



    {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







    share|improve this answer











    $endgroup$













    • $begingroup$
      Henrik: Can I add the S in front of the result; e.g., (S1,S8),(S3,S4),(S5,S6,S7)?
      $endgroup$
      – PRG
      Apr 25 at 19:26










    • $begingroup$
      MANY THANKS, HENRIK!
      $endgroup$
      – PRG
      Apr 25 at 20:33










    • $begingroup$
      YOU'RE WELCOME, PRG! =D
      $endgroup$
      – Henrik Schumacher
      Apr 25 at 20:34



















    6












    $begingroup$

    The function positionDuplicates from How to efficiently find positions of duplicates? does the job, faster than Nearest.



    (* Henrik's method *)
    posDupes[M_] := DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]

    SeedRandom[0]; (* to make a reproducible 1000 x 1000 matrix *)
    foo = Nest[RandomInteger[1, {1000, 1000}] # &, 1, 9];

    d1 = Cases[positionDuplicates[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
    (* {0.017, Null} *)

    d2 = Cases[posDupes[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
    (* {0.060, Null} *)

    d1 == d2
    (* True *)

    d1
    (*
    {{25, 75, 291, 355, 356, 425, 475, 518, 547, 668, 670, 750, 777},
    {173, 516}, {544, 816}, {610, 720}}
    *)





    share|improve this answer









    $endgroup$









    • 1




      $begingroup$
      Cases[Values[PositionIndex[M]], dupe_ /; Length[dupe] > 1] is faster than positionDuplicates
      $endgroup$
      – Okkes Dulgerci
      Apr 26 at 1:41










    • $begingroup$
      @OkkesDulgerci Yes, it is for me, too, in V12. My main point is that the solution to this question has been given in another Q&A. See this answer for the PositionIndex solutoin.
      $endgroup$
      – Michael E2
      Apr 26 at 1:44








    • 3




      $begingroup$
      @OkkesDulgerci It's interesting that PositionIndex outperforms positionDuplicates on a list of lists, because it is still much slower on a list of integers.
      $endgroup$
      – Michael E2
      Apr 26 at 1:53



















    3












    $begingroup$

    While this question repeats a previous query about finding DuplicatePositions, the duplicates here are amongst a list of binary vectors in contrast to the original duplicates occurring amongst a list of numbers. As illustrated in an answer to the original query however, the type, depth and distribution of inputs can significantly impact efficiency so there may well be further optimizations specific to this case of finding duplicates amongst binary vectors. The following summarises timings of the "superfunction" DuplicatePositions (collected and defined from answers to the original question - in particular Szabolcs, Carl Woll and Mr.Wizard), postionDuplicates (the fastest solutions for numbers from Szabolcs) and a tweeking in the "UseGatherByLocalMap" Method option (from Carl Woll), the accepted groupBy answer (by Roman) and the nearest answer (by Henrik Schumacher) for various types of binary vectors. I've contributed the "UseOrdering" Method in DuplicatePositions.



    duplicatePositionsByOrdering[ls_]:= SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First]


    which seems to do well for sparse vectors (a more succinct version of similar ideas used by Mr.Wizard and Leonid Shifrin in their anwers). Note that a random 1000x1000 binary matrix is very likely to be sparse to the point of no (row) duplicates occurring so presumably in the OP's situation the authentic data is not randomly generated and instead includes manufactured repeats. To the timings (the tag function just puts in the S1, S2 ... tags as originally requested and the tick indicates identical output):



    enter image description here



    Obviously timings aren't everything as short-clear functions can often be preferable (as well as potentially being more efficient for different inputs) but it can also sometimes be illuminating--here for example, indicating that GroupBy seems to recognize order for ragged vectors unlike GatherBy.



    The code for the above output is below



    SetAttributes[benchmark, HoldAll];

    benchmark[functions_, opts : OptionsPattern] :=
    Function[input, benchmark[functions, input, opts], HoldAll];

    benchmark[functions_, input_, OptionsPattern] := Module[{
    tm = Function[fn,
    Function[x, <|ToString[fn] -> RepeatedTiming@fn@x|>]],
    timesOutputs, times},
    SeedRandom@0;
    timesOutputs = Through[(tm /@ functions)@input];
    times =
    SortBy[Query[All, All, First]@timesOutputs, Last] // Dataset;
    If[OptionValue@"CheckOutputs",
    Labeled[times,
    Row[{ToString@Unevaluated@input, Spacer@80,
    If[SameQ @@ (Query[All, Last, 2]@timesOutputs),
    Style["[Checkmark]", Green, 20], Style["x", Red, 20]]}],
    Top], times]
    ];

    Options[benchmark] = {"CheckOutputs" -> True};

    Options[DuplicatePositions] = {Method -> Automatic};

    DuplicatePositions[ls_, OptionsPattern] :=
    With[{method = OptionValue[Method]},
    Switch[method,
    "UseGatherBy", GatherBy[Range@Length@ls, ls[[#]] &],
    "UsePositionIndex", Values@PositionIndex@ls,
    "UseOrdering", SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First],
    "UseGatherByLocalMap", Module[{func}, func /: Map[func, _] := ls;
    GatherBy[Range@Length@ls, func]],
    Automatic, Which[
    ArrayQ[ls, 1, NumericQ],
    DuplicatePositions[ls, "Method" -> "UseGatherBy" ],
    ArrayQ[ls, 2, NumericQ], DuplicatePositionsBy[ls, FromDigits],
    MatchQ[{{_?IntegerQ ..} ..}]@ls,
    DuplicatePositionsBy[ls, FromDigits],
    True, DuplicatePositions[ls, Method -> "UsePositionIndex" ]
    ]]];

    DuplicatePositionsBy[ls_, fn_, opts : OptionsPattern] :=
    DuplicatePositions[fn /@ ls, opts];

    tag = Map["S" <> ToString@# &, #, {-1}] &;
    positionDuplicates[ls_] := GatherBy[Range@Length@ls, ls[[#]] &];
    groupBy[M_] := With[
    {rownames = Array["S" <> ToString[#] &, Length[M]]},
    Values@GroupBy[Thread[rownames -> M], Last -> First]];
    nearest[M_] :=
    DeleteDuplicates[
    Sort /@ Nearest[M -> Automatic, M, {[Infinity], 0}]];
    n = 10^4;
    binaryVectors50k =
    IntegerDigits[#, 2, 13] & /@ RandomInteger[n, 5*n];
    fns = {
    groupBy,
    (nearest@# // tag) &,
    (DuplicatePositions@# // tag) &,
    (DuplicatePositionsBy[#, FromDigits[#, 2] &,
    Method -> "UseGatherByLocalMap"] // tag) &,
    (positionDuplicates@# // tag) &
    };
    benchmark[fns]@binaryVectors50k
    n = 10^3;
    binaryVectorsRagged5k = IntegerDigits[#, 2] & /@ RandomInteger[n, 5*n];
    fns = {
    groupBy,
    (DuplicatePositions@# // tag) &,
    (DuplicatePositionsBy[#, FromDigits[#, 2] &,
    Method -> "UseGatherByLocalMap"] // tag) &,
    (positionDuplicates@# // tag) &
    };
    benchmark[fns]@binaryVectorsRagged5k

    n = 10^4;
    binaryVectorsSparse10k := RandomInteger[1, {n, n}];
    fns = {
    (DuplicatePositions@# // tag) &,
    (positionDuplicates@# // tag) &,
    (DuplicatePositions[#, Method -> "UseOrdering"] // tag) &,
    groupBy};
    benchmark[fns]@binaryVectorsSparse10k





    share|improve this answer











    $endgroup$




















      4 Answers
      4






      active

      oldest

      votes








      4 Answers
      4






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      2












      $begingroup$

      I'd use GroupBy.



      First the names of the rows: can be anything you like, for example



      rownames = Array[ToExpression["S" <> ToString[#]] &, Length[M]]



      {S1, S2, S3, S4, S5, S6, S7, S8}




      Next the grouping:



      groups = GroupBy[Thread[rownames -> M], Last -> First]



      <|{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} -> {S1, S8},
      {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0} -> {S2, S3, S4},
      {0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0} -> {S5, S6, S7}|>




      If all you need are the names:



      Values[groups]



      {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







      share|improve this answer









      $endgroup$


















        2












        $begingroup$

        I'd use GroupBy.



        First the names of the rows: can be anything you like, for example



        rownames = Array[ToExpression["S" <> ToString[#]] &, Length[M]]



        {S1, S2, S3, S4, S5, S6, S7, S8}




        Next the grouping:



        groups = GroupBy[Thread[rownames -> M], Last -> First]



        <|{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} -> {S1, S8},
        {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0} -> {S2, S3, S4},
        {0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0} -> {S5, S6, S7}|>




        If all you need are the names:



        Values[groups]



        {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







        share|improve this answer









        $endgroup$
















          2












          2








          2





          $begingroup$

          I'd use GroupBy.



          First the names of the rows: can be anything you like, for example



          rownames = Array[ToExpression["S" <> ToString[#]] &, Length[M]]



          {S1, S2, S3, S4, S5, S6, S7, S8}




          Next the grouping:



          groups = GroupBy[Thread[rownames -> M], Last -> First]



          <|{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} -> {S1, S8},
          {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0} -> {S2, S3, S4},
          {0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0} -> {S5, S6, S7}|>




          If all you need are the names:



          Values[groups]



          {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







          share|improve this answer









          $endgroup$



          I'd use GroupBy.



          First the names of the rows: can be anything you like, for example



          rownames = Array[ToExpression["S" <> ToString[#]] &, Length[M]]



          {S1, S2, S3, S4, S5, S6, S7, S8}




          Next the grouping:



          groups = GroupBy[Thread[rownames -> M], Last -> First]



          <|{0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} -> {S1, S8},
          {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0} -> {S2, S3, S4},
          {0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0} -> {S5, S6, S7}|>




          If all you need are the names:



          Values[groups]



          {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}








          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Apr 26 at 11:45









          RomanRoman

          9,59511440




          9,59511440























              8












              $begingroup$

              idx = DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]



              {{1, 8}, {2, 3, 4}, {5, 6, 7}}




              In order to obtain the labels of the rows, you may use the following:



              labels = {S1, S2, S3, S4, S5, S6, S7, S8};
              Map[labels[[#]] &, idx, {2}]



              {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







              share|improve this answer











              $endgroup$













              • $begingroup$
                Henrik: Can I add the S in front of the result; e.g., (S1,S8),(S3,S4),(S5,S6,S7)?
                $endgroup$
                – PRG
                Apr 25 at 19:26










              • $begingroup$
                MANY THANKS, HENRIK!
                $endgroup$
                – PRG
                Apr 25 at 20:33










              • $begingroup$
                YOU'RE WELCOME, PRG! =D
                $endgroup$
                – Henrik Schumacher
                Apr 25 at 20:34
















              8












              $begingroup$

              idx = DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]



              {{1, 8}, {2, 3, 4}, {5, 6, 7}}




              In order to obtain the labels of the rows, you may use the following:



              labels = {S1, S2, S3, S4, S5, S6, S7, S8};
              Map[labels[[#]] &, idx, {2}]



              {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







              share|improve this answer











              $endgroup$













              • $begingroup$
                Henrik: Can I add the S in front of the result; e.g., (S1,S8),(S3,S4),(S5,S6,S7)?
                $endgroup$
                – PRG
                Apr 25 at 19:26










              • $begingroup$
                MANY THANKS, HENRIK!
                $endgroup$
                – PRG
                Apr 25 at 20:33










              • $begingroup$
                YOU'RE WELCOME, PRG! =D
                $endgroup$
                – Henrik Schumacher
                Apr 25 at 20:34














              8












              8








              8





              $begingroup$

              idx = DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]



              {{1, 8}, {2, 3, 4}, {5, 6, 7}}




              In order to obtain the labels of the rows, you may use the following:



              labels = {S1, S2, S3, S4, S5, S6, S7, S8};
              Map[labels[[#]] &, idx, {2}]



              {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}







              share|improve this answer











              $endgroup$



              idx = DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]



              {{1, 8}, {2, 3, 4}, {5, 6, 7}}




              In order to obtain the labels of the rows, you may use the following:



              labels = {S1, S2, S3, S4, S5, S6, S7, S8};
              Map[labels[[#]] &, idx, {2}]



              {{S1, S8}, {S2, S3, S4}, {S5, S6, S7}}








              share|improve this answer














              share|improve this answer



              share|improve this answer








              edited Apr 25 at 20:04

























              answered Apr 25 at 19:22









              Henrik SchumacherHenrik Schumacher

              63.2k587176




              63.2k587176












              • $begingroup$
                Henrik: Can I add the S in front of the result; e.g., (S1,S8),(S3,S4),(S5,S6,S7)?
                $endgroup$
                – PRG
                Apr 25 at 19:26










              • $begingroup$
                MANY THANKS, HENRIK!
                $endgroup$
                – PRG
                Apr 25 at 20:33










              • $begingroup$
                YOU'RE WELCOME, PRG! =D
                $endgroup$
                – Henrik Schumacher
                Apr 25 at 20:34


















              • $begingroup$
                Henrik: Can I add the S in front of the result; e.g., (S1,S8),(S3,S4),(S5,S6,S7)?
                $endgroup$
                – PRG
                Apr 25 at 19:26










              • $begingroup$
                MANY THANKS, HENRIK!
                $endgroup$
                – PRG
                Apr 25 at 20:33










              • $begingroup$
                YOU'RE WELCOME, PRG! =D
                $endgroup$
                – Henrik Schumacher
                Apr 25 at 20:34
















              $begingroup$
              Henrik: Can I add the S in front of the result; e.g., (S1,S8),(S3,S4),(S5,S6,S7)?
              $endgroup$
              – PRG
              Apr 25 at 19:26




              $begingroup$
              Henrik: Can I add the S in front of the result; e.g., (S1,S8),(S3,S4),(S5,S6,S7)?
              $endgroup$
              – PRG
              Apr 25 at 19:26












              $begingroup$
              MANY THANKS, HENRIK!
              $endgroup$
              – PRG
              Apr 25 at 20:33




              $begingroup$
              MANY THANKS, HENRIK!
              $endgroup$
              – PRG
              Apr 25 at 20:33












              $begingroup$
              YOU'RE WELCOME, PRG! =D
              $endgroup$
              – Henrik Schumacher
              Apr 25 at 20:34




              $begingroup$
              YOU'RE WELCOME, PRG! =D
              $endgroup$
              – Henrik Schumacher
              Apr 25 at 20:34











              6












              $begingroup$

              The function positionDuplicates from How to efficiently find positions of duplicates? does the job, faster than Nearest.



              (* Henrik's method *)
              posDupes[M_] := DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]

              SeedRandom[0]; (* to make a reproducible 1000 x 1000 matrix *)
              foo = Nest[RandomInteger[1, {1000, 1000}] # &, 1, 9];

              d1 = Cases[positionDuplicates[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.017, Null} *)

              d2 = Cases[posDupes[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.060, Null} *)

              d1 == d2
              (* True *)

              d1
              (*
              {{25, 75, 291, 355, 356, 425, 475, 518, 547, 668, 670, 750, 777},
              {173, 516}, {544, 816}, {610, 720}}
              *)





              share|improve this answer









              $endgroup$









              • 1




                $begingroup$
                Cases[Values[PositionIndex[M]], dupe_ /; Length[dupe] > 1] is faster than positionDuplicates
                $endgroup$
                – Okkes Dulgerci
                Apr 26 at 1:41










              • $begingroup$
                @OkkesDulgerci Yes, it is for me, too, in V12. My main point is that the solution to this question has been given in another Q&A. See this answer for the PositionIndex solutoin.
                $endgroup$
                – Michael E2
                Apr 26 at 1:44








              • 3




                $begingroup$
                @OkkesDulgerci It's interesting that PositionIndex outperforms positionDuplicates on a list of lists, because it is still much slower on a list of integers.
                $endgroup$
                – Michael E2
                Apr 26 at 1:53
















              6












              $begingroup$

              The function positionDuplicates from How to efficiently find positions of duplicates? does the job, faster than Nearest.



              (* Henrik's method *)
              posDupes[M_] := DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]

              SeedRandom[0]; (* to make a reproducible 1000 x 1000 matrix *)
              foo = Nest[RandomInteger[1, {1000, 1000}] # &, 1, 9];

              d1 = Cases[positionDuplicates[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.017, Null} *)

              d2 = Cases[posDupes[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.060, Null} *)

              d1 == d2
              (* True *)

              d1
              (*
              {{25, 75, 291, 355, 356, 425, 475, 518, 547, 668, 670, 750, 777},
              {173, 516}, {544, 816}, {610, 720}}
              *)





              share|improve this answer









              $endgroup$









              • 1




                $begingroup$
                Cases[Values[PositionIndex[M]], dupe_ /; Length[dupe] > 1] is faster than positionDuplicates
                $endgroup$
                – Okkes Dulgerci
                Apr 26 at 1:41










              • $begingroup$
                @OkkesDulgerci Yes, it is for me, too, in V12. My main point is that the solution to this question has been given in another Q&A. See this answer for the PositionIndex solutoin.
                $endgroup$
                – Michael E2
                Apr 26 at 1:44








              • 3




                $begingroup$
                @OkkesDulgerci It's interesting that PositionIndex outperforms positionDuplicates on a list of lists, because it is still much slower on a list of integers.
                $endgroup$
                – Michael E2
                Apr 26 at 1:53














              6












              6








              6





              $begingroup$

              The function positionDuplicates from How to efficiently find positions of duplicates? does the job, faster than Nearest.



              (* Henrik's method *)
              posDupes[M_] := DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]

              SeedRandom[0]; (* to make a reproducible 1000 x 1000 matrix *)
              foo = Nest[RandomInteger[1, {1000, 1000}] # &, 1, 9];

              d1 = Cases[positionDuplicates[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.017, Null} *)

              d2 = Cases[posDupes[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.060, Null} *)

              d1 == d2
              (* True *)

              d1
              (*
              {{25, 75, 291, 355, 356, 425, 475, 518, 547, 668, 670, 750, 777},
              {173, 516}, {544, 816}, {610, 720}}
              *)





              share|improve this answer









              $endgroup$



              The function positionDuplicates from How to efficiently find positions of duplicates? does the job, faster than Nearest.



              (* Henrik's method *)
              posDupes[M_] := DeleteDuplicates[Sort /@ Nearest[M -> Automatic, M, {∞, 0}]]

              SeedRandom[0]; (* to make a reproducible 1000 x 1000 matrix *)
              foo = Nest[RandomInteger[1, {1000, 1000}] # &, 1, 9];

              d1 = Cases[positionDuplicates[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.017, Null} *)

              d2 = Cases[posDupes[foo], dupe_ /; Length[dupe] > 1]; // RepeatedTiming
              (* {0.060, Null} *)

              d1 == d2
              (* True *)

              d1
              (*
              {{25, 75, 291, 355, 356, 425, 475, 518, 547, 668, 670, 750, 777},
              {173, 516}, {544, 816}, {610, 720}}
              *)






              share|improve this answer












              share|improve this answer



              share|improve this answer










              answered Apr 26 at 0:28









              Michael E2Michael E2

              153k12208495




              153k12208495








              • 1




                $begingroup$
                Cases[Values[PositionIndex[M]], dupe_ /; Length[dupe] > 1] is faster than positionDuplicates
                $endgroup$
                – Okkes Dulgerci
                Apr 26 at 1:41










              • $begingroup$
                @OkkesDulgerci Yes, it is for me, too, in V12. My main point is that the solution to this question has been given in another Q&A. See this answer for the PositionIndex solutoin.
                $endgroup$
                – Michael E2
                Apr 26 at 1:44








              • 3




                $begingroup$
                @OkkesDulgerci It's interesting that PositionIndex outperforms positionDuplicates on a list of lists, because it is still much slower on a list of integers.
                $endgroup$
                – Michael E2
                Apr 26 at 1:53














              • 1




                $begingroup$
                Cases[Values[PositionIndex[M]], dupe_ /; Length[dupe] > 1] is faster than positionDuplicates
                $endgroup$
                – Okkes Dulgerci
                Apr 26 at 1:41










              • $begingroup$
                @OkkesDulgerci Yes, it is for me, too, in V12. My main point is that the solution to this question has been given in another Q&A. See this answer for the PositionIndex solutoin.
                $endgroup$
                – Michael E2
                Apr 26 at 1:44








              • 3




                $begingroup$
                @OkkesDulgerci It's interesting that PositionIndex outperforms positionDuplicates on a list of lists, because it is still much slower on a list of integers.
                $endgroup$
                – Michael E2
                Apr 26 at 1:53








              1




              1




              $begingroup$
              Cases[Values[PositionIndex[M]], dupe_ /; Length[dupe] > 1] is faster than positionDuplicates
              $endgroup$
              – Okkes Dulgerci
              Apr 26 at 1:41




              $begingroup$
              Cases[Values[PositionIndex[M]], dupe_ /; Length[dupe] > 1] is faster than positionDuplicates
              $endgroup$
              – Okkes Dulgerci
              Apr 26 at 1:41












              $begingroup$
              @OkkesDulgerci Yes, it is for me, too, in V12. My main point is that the solution to this question has been given in another Q&A. See this answer for the PositionIndex solutoin.
              $endgroup$
              – Michael E2
              Apr 26 at 1:44






              $begingroup$
              @OkkesDulgerci Yes, it is for me, too, in V12. My main point is that the solution to this question has been given in another Q&A. See this answer for the PositionIndex solutoin.
              $endgroup$
              – Michael E2
              Apr 26 at 1:44






              3




              3




              $begingroup$
              @OkkesDulgerci It's interesting that PositionIndex outperforms positionDuplicates on a list of lists, because it is still much slower on a list of integers.
              $endgroup$
              – Michael E2
              Apr 26 at 1:53




              $begingroup$
              @OkkesDulgerci It's interesting that PositionIndex outperforms positionDuplicates on a list of lists, because it is still much slower on a list of integers.
              $endgroup$
              – Michael E2
              Apr 26 at 1:53











              3












              $begingroup$

              While this question repeats a previous query about finding DuplicatePositions, the duplicates here are amongst a list of binary vectors in contrast to the original duplicates occurring amongst a list of numbers. As illustrated in an answer to the original query however, the type, depth and distribution of inputs can significantly impact efficiency so there may well be further optimizations specific to this case of finding duplicates amongst binary vectors. The following summarises timings of the "superfunction" DuplicatePositions (collected and defined from answers to the original question - in particular Szabolcs, Carl Woll and Mr.Wizard), postionDuplicates (the fastest solutions for numbers from Szabolcs) and a tweeking in the "UseGatherByLocalMap" Method option (from Carl Woll), the accepted groupBy answer (by Roman) and the nearest answer (by Henrik Schumacher) for various types of binary vectors. I've contributed the "UseOrdering" Method in DuplicatePositions.



              duplicatePositionsByOrdering[ls_]:= SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First]


              which seems to do well for sparse vectors (a more succinct version of similar ideas used by Mr.Wizard and Leonid Shifrin in their anwers). Note that a random 1000x1000 binary matrix is very likely to be sparse to the point of no (row) duplicates occurring so presumably in the OP's situation the authentic data is not randomly generated and instead includes manufactured repeats. To the timings (the tag function just puts in the S1, S2 ... tags as originally requested and the tick indicates identical output):



              enter image description here



              Obviously timings aren't everything as short-clear functions can often be preferable (as well as potentially being more efficient for different inputs) but it can also sometimes be illuminating--here for example, indicating that GroupBy seems to recognize order for ragged vectors unlike GatherBy.



              The code for the above output is below



              SetAttributes[benchmark, HoldAll];

              benchmark[functions_, opts : OptionsPattern] :=
              Function[input, benchmark[functions, input, opts], HoldAll];

              benchmark[functions_, input_, OptionsPattern] := Module[{
              tm = Function[fn,
              Function[x, <|ToString[fn] -> RepeatedTiming@fn@x|>]],
              timesOutputs, times},
              SeedRandom@0;
              timesOutputs = Through[(tm /@ functions)@input];
              times =
              SortBy[Query[All, All, First]@timesOutputs, Last] // Dataset;
              If[OptionValue@"CheckOutputs",
              Labeled[times,
              Row[{ToString@Unevaluated@input, Spacer@80,
              If[SameQ @@ (Query[All, Last, 2]@timesOutputs),
              Style["[Checkmark]", Green, 20], Style["x", Red, 20]]}],
              Top], times]
              ];

              Options[benchmark] = {"CheckOutputs" -> True};

              Options[DuplicatePositions] = {Method -> Automatic};

              DuplicatePositions[ls_, OptionsPattern] :=
              With[{method = OptionValue[Method]},
              Switch[method,
              "UseGatherBy", GatherBy[Range@Length@ls, ls[[#]] &],
              "UsePositionIndex", Values@PositionIndex@ls,
              "UseOrdering", SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First],
              "UseGatherByLocalMap", Module[{func}, func /: Map[func, _] := ls;
              GatherBy[Range@Length@ls, func]],
              Automatic, Which[
              ArrayQ[ls, 1, NumericQ],
              DuplicatePositions[ls, "Method" -> "UseGatherBy" ],
              ArrayQ[ls, 2, NumericQ], DuplicatePositionsBy[ls, FromDigits],
              MatchQ[{{_?IntegerQ ..} ..}]@ls,
              DuplicatePositionsBy[ls, FromDigits],
              True, DuplicatePositions[ls, Method -> "UsePositionIndex" ]
              ]]];

              DuplicatePositionsBy[ls_, fn_, opts : OptionsPattern] :=
              DuplicatePositions[fn /@ ls, opts];

              tag = Map["S" <> ToString@# &, #, {-1}] &;
              positionDuplicates[ls_] := GatherBy[Range@Length@ls, ls[[#]] &];
              groupBy[M_] := With[
              {rownames = Array["S" <> ToString[#] &, Length[M]]},
              Values@GroupBy[Thread[rownames -> M], Last -> First]];
              nearest[M_] :=
              DeleteDuplicates[
              Sort /@ Nearest[M -> Automatic, M, {[Infinity], 0}]];
              n = 10^4;
              binaryVectors50k =
              IntegerDigits[#, 2, 13] & /@ RandomInteger[n, 5*n];
              fns = {
              groupBy,
              (nearest@# // tag) &,
              (DuplicatePositions@# // tag) &,
              (DuplicatePositionsBy[#, FromDigits[#, 2] &,
              Method -> "UseGatherByLocalMap"] // tag) &,
              (positionDuplicates@# // tag) &
              };
              benchmark[fns]@binaryVectors50k
              n = 10^3;
              binaryVectorsRagged5k = IntegerDigits[#, 2] & /@ RandomInteger[n, 5*n];
              fns = {
              groupBy,
              (DuplicatePositions@# // tag) &,
              (DuplicatePositionsBy[#, FromDigits[#, 2] &,
              Method -> "UseGatherByLocalMap"] // tag) &,
              (positionDuplicates@# // tag) &
              };
              benchmark[fns]@binaryVectorsRagged5k

              n = 10^4;
              binaryVectorsSparse10k := RandomInteger[1, {n, n}];
              fns = {
              (DuplicatePositions@# // tag) &,
              (positionDuplicates@# // tag) &,
              (DuplicatePositions[#, Method -> "UseOrdering"] // tag) &,
              groupBy};
              benchmark[fns]@binaryVectorsSparse10k





              share|improve this answer











              $endgroup$


















                3












                $begingroup$

                While this question repeats a previous query about finding DuplicatePositions, the duplicates here are amongst a list of binary vectors in contrast to the original duplicates occurring amongst a list of numbers. As illustrated in an answer to the original query however, the type, depth and distribution of inputs can significantly impact efficiency so there may well be further optimizations specific to this case of finding duplicates amongst binary vectors. The following summarises timings of the "superfunction" DuplicatePositions (collected and defined from answers to the original question - in particular Szabolcs, Carl Woll and Mr.Wizard), postionDuplicates (the fastest solutions for numbers from Szabolcs) and a tweeking in the "UseGatherByLocalMap" Method option (from Carl Woll), the accepted groupBy answer (by Roman) and the nearest answer (by Henrik Schumacher) for various types of binary vectors. I've contributed the "UseOrdering" Method in DuplicatePositions.



                duplicatePositionsByOrdering[ls_]:= SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First]


                which seems to do well for sparse vectors (a more succinct version of similar ideas used by Mr.Wizard and Leonid Shifrin in their anwers). Note that a random 1000x1000 binary matrix is very likely to be sparse to the point of no (row) duplicates occurring so presumably in the OP's situation the authentic data is not randomly generated and instead includes manufactured repeats. To the timings (the tag function just puts in the S1, S2 ... tags as originally requested and the tick indicates identical output):



                enter image description here



                Obviously timings aren't everything as short-clear functions can often be preferable (as well as potentially being more efficient for different inputs) but it can also sometimes be illuminating--here for example, indicating that GroupBy seems to recognize order for ragged vectors unlike GatherBy.



                The code for the above output is below



                SetAttributes[benchmark, HoldAll];

                benchmark[functions_, opts : OptionsPattern] :=
                Function[input, benchmark[functions, input, opts], HoldAll];

                benchmark[functions_, input_, OptionsPattern] := Module[{
                tm = Function[fn,
                Function[x, <|ToString[fn] -> RepeatedTiming@fn@x|>]],
                timesOutputs, times},
                SeedRandom@0;
                timesOutputs = Through[(tm /@ functions)@input];
                times =
                SortBy[Query[All, All, First]@timesOutputs, Last] // Dataset;
                If[OptionValue@"CheckOutputs",
                Labeled[times,
                Row[{ToString@Unevaluated@input, Spacer@80,
                If[SameQ @@ (Query[All, Last, 2]@timesOutputs),
                Style["[Checkmark]", Green, 20], Style["x", Red, 20]]}],
                Top], times]
                ];

                Options[benchmark] = {"CheckOutputs" -> True};

                Options[DuplicatePositions] = {Method -> Automatic};

                DuplicatePositions[ls_, OptionsPattern] :=
                With[{method = OptionValue[Method]},
                Switch[method,
                "UseGatherBy", GatherBy[Range@Length@ls, ls[[#]] &],
                "UsePositionIndex", Values@PositionIndex@ls,
                "UseOrdering", SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First],
                "UseGatherByLocalMap", Module[{func}, func /: Map[func, _] := ls;
                GatherBy[Range@Length@ls, func]],
                Automatic, Which[
                ArrayQ[ls, 1, NumericQ],
                DuplicatePositions[ls, "Method" -> "UseGatherBy" ],
                ArrayQ[ls, 2, NumericQ], DuplicatePositionsBy[ls, FromDigits],
                MatchQ[{{_?IntegerQ ..} ..}]@ls,
                DuplicatePositionsBy[ls, FromDigits],
                True, DuplicatePositions[ls, Method -> "UsePositionIndex" ]
                ]]];

                DuplicatePositionsBy[ls_, fn_, opts : OptionsPattern] :=
                DuplicatePositions[fn /@ ls, opts];

                tag = Map["S" <> ToString@# &, #, {-1}] &;
                positionDuplicates[ls_] := GatherBy[Range@Length@ls, ls[[#]] &];
                groupBy[M_] := With[
                {rownames = Array["S" <> ToString[#] &, Length[M]]},
                Values@GroupBy[Thread[rownames -> M], Last -> First]];
                nearest[M_] :=
                DeleteDuplicates[
                Sort /@ Nearest[M -> Automatic, M, {[Infinity], 0}]];
                n = 10^4;
                binaryVectors50k =
                IntegerDigits[#, 2, 13] & /@ RandomInteger[n, 5*n];
                fns = {
                groupBy,
                (nearest@# // tag) &,
                (DuplicatePositions@# // tag) &,
                (DuplicatePositionsBy[#, FromDigits[#, 2] &,
                Method -> "UseGatherByLocalMap"] // tag) &,
                (positionDuplicates@# // tag) &
                };
                benchmark[fns]@binaryVectors50k
                n = 10^3;
                binaryVectorsRagged5k = IntegerDigits[#, 2] & /@ RandomInteger[n, 5*n];
                fns = {
                groupBy,
                (DuplicatePositions@# // tag) &,
                (DuplicatePositionsBy[#, FromDigits[#, 2] &,
                Method -> "UseGatherByLocalMap"] // tag) &,
                (positionDuplicates@# // tag) &
                };
                benchmark[fns]@binaryVectorsRagged5k

                n = 10^4;
                binaryVectorsSparse10k := RandomInteger[1, {n, n}];
                fns = {
                (DuplicatePositions@# // tag) &,
                (positionDuplicates@# // tag) &,
                (DuplicatePositions[#, Method -> "UseOrdering"] // tag) &,
                groupBy};
                benchmark[fns]@binaryVectorsSparse10k





                share|improve this answer











                $endgroup$
















                  3












                  3








                  3





                  $begingroup$

                  While this question repeats a previous query about finding DuplicatePositions, the duplicates here are amongst a list of binary vectors in contrast to the original duplicates occurring amongst a list of numbers. As illustrated in an answer to the original query however, the type, depth and distribution of inputs can significantly impact efficiency so there may well be further optimizations specific to this case of finding duplicates amongst binary vectors. The following summarises timings of the "superfunction" DuplicatePositions (collected and defined from answers to the original question - in particular Szabolcs, Carl Woll and Mr.Wizard), postionDuplicates (the fastest solutions for numbers from Szabolcs) and a tweeking in the "UseGatherByLocalMap" Method option (from Carl Woll), the accepted groupBy answer (by Roman) and the nearest answer (by Henrik Schumacher) for various types of binary vectors. I've contributed the "UseOrdering" Method in DuplicatePositions.



                  duplicatePositionsByOrdering[ls_]:= SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First]


                  which seems to do well for sparse vectors (a more succinct version of similar ideas used by Mr.Wizard and Leonid Shifrin in their anwers). Note that a random 1000x1000 binary matrix is very likely to be sparse to the point of no (row) duplicates occurring so presumably in the OP's situation the authentic data is not randomly generated and instead includes manufactured repeats. To the timings (the tag function just puts in the S1, S2 ... tags as originally requested and the tick indicates identical output):



                  enter image description here



                  Obviously timings aren't everything as short-clear functions can often be preferable (as well as potentially being more efficient for different inputs) but it can also sometimes be illuminating--here for example, indicating that GroupBy seems to recognize order for ragged vectors unlike GatherBy.



                  The code for the above output is below



                  SetAttributes[benchmark, HoldAll];

                  benchmark[functions_, opts : OptionsPattern] :=
                  Function[input, benchmark[functions, input, opts], HoldAll];

                  benchmark[functions_, input_, OptionsPattern] := Module[{
                  tm = Function[fn,
                  Function[x, <|ToString[fn] -> RepeatedTiming@fn@x|>]],
                  timesOutputs, times},
                  SeedRandom@0;
                  timesOutputs = Through[(tm /@ functions)@input];
                  times =
                  SortBy[Query[All, All, First]@timesOutputs, Last] // Dataset;
                  If[OptionValue@"CheckOutputs",
                  Labeled[times,
                  Row[{ToString@Unevaluated@input, Spacer@80,
                  If[SameQ @@ (Query[All, Last, 2]@timesOutputs),
                  Style["[Checkmark]", Green, 20], Style["x", Red, 20]]}],
                  Top], times]
                  ];

                  Options[benchmark] = {"CheckOutputs" -> True};

                  Options[DuplicatePositions] = {Method -> Automatic};

                  DuplicatePositions[ls_, OptionsPattern] :=
                  With[{method = OptionValue[Method]},
                  Switch[method,
                  "UseGatherBy", GatherBy[Range@Length@ls, ls[[#]] &],
                  "UsePositionIndex", Values@PositionIndex@ls,
                  "UseOrdering", SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First],
                  "UseGatherByLocalMap", Module[{func}, func /: Map[func, _] := ls;
                  GatherBy[Range@Length@ls, func]],
                  Automatic, Which[
                  ArrayQ[ls, 1, NumericQ],
                  DuplicatePositions[ls, "Method" -> "UseGatherBy" ],
                  ArrayQ[ls, 2, NumericQ], DuplicatePositionsBy[ls, FromDigits],
                  MatchQ[{{_?IntegerQ ..} ..}]@ls,
                  DuplicatePositionsBy[ls, FromDigits],
                  True, DuplicatePositions[ls, Method -> "UsePositionIndex" ]
                  ]]];

                  DuplicatePositionsBy[ls_, fn_, opts : OptionsPattern] :=
                  DuplicatePositions[fn /@ ls, opts];

                  tag = Map["S" <> ToString@# &, #, {-1}] &;
                  positionDuplicates[ls_] := GatherBy[Range@Length@ls, ls[[#]] &];
                  groupBy[M_] := With[
                  {rownames = Array["S" <> ToString[#] &, Length[M]]},
                  Values@GroupBy[Thread[rownames -> M], Last -> First]];
                  nearest[M_] :=
                  DeleteDuplicates[
                  Sort /@ Nearest[M -> Automatic, M, {[Infinity], 0}]];
                  n = 10^4;
                  binaryVectors50k =
                  IntegerDigits[#, 2, 13] & /@ RandomInteger[n, 5*n];
                  fns = {
                  groupBy,
                  (nearest@# // tag) &,
                  (DuplicatePositions@# // tag) &,
                  (DuplicatePositionsBy[#, FromDigits[#, 2] &,
                  Method -> "UseGatherByLocalMap"] // tag) &,
                  (positionDuplicates@# // tag) &
                  };
                  benchmark[fns]@binaryVectors50k
                  n = 10^3;
                  binaryVectorsRagged5k = IntegerDigits[#, 2] & /@ RandomInteger[n, 5*n];
                  fns = {
                  groupBy,
                  (DuplicatePositions@# // tag) &,
                  (DuplicatePositionsBy[#, FromDigits[#, 2] &,
                  Method -> "UseGatherByLocalMap"] // tag) &,
                  (positionDuplicates@# // tag) &
                  };
                  benchmark[fns]@binaryVectorsRagged5k

                  n = 10^4;
                  binaryVectorsSparse10k := RandomInteger[1, {n, n}];
                  fns = {
                  (DuplicatePositions@# // tag) &,
                  (positionDuplicates@# // tag) &,
                  (DuplicatePositions[#, Method -> "UseOrdering"] // tag) &,
                  groupBy};
                  benchmark[fns]@binaryVectorsSparse10k





                  share|improve this answer











                  $endgroup$



                  While this question repeats a previous query about finding DuplicatePositions, the duplicates here are amongst a list of binary vectors in contrast to the original duplicates occurring amongst a list of numbers. As illustrated in an answer to the original query however, the type, depth and distribution of inputs can significantly impact efficiency so there may well be further optimizations specific to this case of finding duplicates amongst binary vectors. The following summarises timings of the "superfunction" DuplicatePositions (collected and defined from answers to the original question - in particular Szabolcs, Carl Woll and Mr.Wizard), postionDuplicates (the fastest solutions for numbers from Szabolcs) and a tweeking in the "UseGatherByLocalMap" Method option (from Carl Woll), the accepted groupBy answer (by Roman) and the nearest answer (by Henrik Schumacher) for various types of binary vectors. I've contributed the "UseOrdering" Method in DuplicatePositions.



                  duplicatePositionsByOrdering[ls_]:= SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First]


                  which seems to do well for sparse vectors (a more succinct version of similar ideas used by Mr.Wizard and Leonid Shifrin in their anwers). Note that a random 1000x1000 binary matrix is very likely to be sparse to the point of no (row) duplicates occurring so presumably in the OP's situation the authentic data is not randomly generated and instead includes manufactured repeats. To the timings (the tag function just puts in the S1, S2 ... tags as originally requested and the tick indicates identical output):



                  enter image description here



                  Obviously timings aren't everything as short-clear functions can often be preferable (as well as potentially being more efficient for different inputs) but it can also sometimes be illuminating--here for example, indicating that GroupBy seems to recognize order for ragged vectors unlike GatherBy.



                  The code for the above output is below



                  SetAttributes[benchmark, HoldAll];

                  benchmark[functions_, opts : OptionsPattern] :=
                  Function[input, benchmark[functions, input, opts], HoldAll];

                  benchmark[functions_, input_, OptionsPattern] := Module[{
                  tm = Function[fn,
                  Function[x, <|ToString[fn] -> RepeatedTiming@fn@x|>]],
                  timesOutputs, times},
                  SeedRandom@0;
                  timesOutputs = Through[(tm /@ functions)@input];
                  times =
                  SortBy[Query[All, All, First]@timesOutputs, Last] // Dataset;
                  If[OptionValue@"CheckOutputs",
                  Labeled[times,
                  Row[{ToString@Unevaluated@input, Spacer@80,
                  If[SameQ @@ (Query[All, Last, 2]@timesOutputs),
                  Style["[Checkmark]", Green, 20], Style["x", Red, 20]]}],
                  Top], times]
                  ];

                  Options[benchmark] = {"CheckOutputs" -> True};

                  Options[DuplicatePositions] = {Method -> Automatic};

                  DuplicatePositions[ls_, OptionsPattern] :=
                  With[{method = OptionValue[Method]},
                  Switch[method,
                  "UseGatherBy", GatherBy[Range@Length@ls, ls[[#]] &],
                  "UsePositionIndex", Values@PositionIndex@ls,
                  "UseOrdering", SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First],
                  "UseGatherByLocalMap", Module[{func}, func /: Map[func, _] := ls;
                  GatherBy[Range@Length@ls, func]],
                  Automatic, Which[
                  ArrayQ[ls, 1, NumericQ],
                  DuplicatePositions[ls, "Method" -> "UseGatherBy" ],
                  ArrayQ[ls, 2, NumericQ], DuplicatePositionsBy[ls, FromDigits],
                  MatchQ[{{_?IntegerQ ..} ..}]@ls,
                  DuplicatePositionsBy[ls, FromDigits],
                  True, DuplicatePositions[ls, Method -> "UsePositionIndex" ]
                  ]]];

                  DuplicatePositionsBy[ls_, fn_, opts : OptionsPattern] :=
                  DuplicatePositions[fn /@ ls, opts];

                  tag = Map["S" <> ToString@# &, #, {-1}] &;
                  positionDuplicates[ls_] := GatherBy[Range@Length@ls, ls[[#]] &];
                  groupBy[M_] := With[
                  {rownames = Array["S" <> ToString[#] &, Length[M]]},
                  Values@GroupBy[Thread[rownames -> M], Last -> First]];
                  nearest[M_] :=
                  DeleteDuplicates[
                  Sort /@ Nearest[M -> Automatic, M, {[Infinity], 0}]];
                  n = 10^4;
                  binaryVectors50k =
                  IntegerDigits[#, 2, 13] & /@ RandomInteger[n, 5*n];
                  fns = {
                  groupBy,
                  (nearest@# // tag) &,
                  (DuplicatePositions@# // tag) &,
                  (DuplicatePositionsBy[#, FromDigits[#, 2] &,
                  Method -> "UseGatherByLocalMap"] // tag) &,
                  (positionDuplicates@# // tag) &
                  };
                  benchmark[fns]@binaryVectors50k
                  n = 10^3;
                  binaryVectorsRagged5k = IntegerDigits[#, 2] & /@ RandomInteger[n, 5*n];
                  fns = {
                  groupBy,
                  (DuplicatePositions@# // tag) &,
                  (DuplicatePositionsBy[#, FromDigits[#, 2] &,
                  Method -> "UseGatherByLocalMap"] // tag) &,
                  (positionDuplicates@# // tag) &
                  };
                  benchmark[fns]@binaryVectorsRagged5k

                  n = 10^4;
                  binaryVectorsSparse10k := RandomInteger[1, {n, n}];
                  fns = {
                  (DuplicatePositions@# // tag) &,
                  (positionDuplicates@# // tag) &,
                  (DuplicatePositions[#, Method -> "UseOrdering"] // tag) &,
                  groupBy};
                  benchmark[fns]@binaryVectorsSparse10k






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited May 8 at 23:08

























                  answered Apr 26 at 13:40









                  Ronald MonsonRonald Monson

                  3,2331735




                  3,2331735















                      Popular posts from this blog

                      Færeyskur hestur Heimild | Tengill | Tilvísanir | LeiðsagnarvalRossið - síða um færeyska hrossið á færeyskuGott ár hjá færeyska hestinum

                      He _____ here since 1970 . Answer needed [closed]What does “since he was so high” mean?Meaning of “catch birds for”?How do I ensure “since” takes the meaning I want?“Who cares here” meaningWhat does “right round toward” mean?the time tense (had now been detected)What does the phrase “ring around the roses” mean here?Correct usage of “visited upon”Meaning of “foiled rail sabotage bid”It was the third time I had gone to Rome or It is the third time I had been to Rome

                      Slayer Innehåll Historia | Stil, komposition och lyrik | Bandets betydelse och framgångar | Sidoprojekt och samarbeten | Kontroverser | Medlemmar | Utmärkelser och nomineringar | Turnéer och festivaler | Diskografi | Referenser | Externa länkar | Navigeringsmenywww.slayer.net”Metal Massacre vol. 1””Metal Massacre vol. 3””Metal Massacre Volume III””Show No Mercy””Haunting the Chapel””Live Undead””Hell Awaits””Reign in Blood””Reign in Blood””Gold & Platinum – Reign in Blood””Golden Gods Awards Winners”originalet”Kerrang! Hall Of Fame””Slayer Looks Back On 37-Year Career In New Video Series: Part Two””South of Heaven””Gold & Platinum – South of Heaven””Seasons in the Abyss””Gold & Platinum - Seasons in the Abyss””Divine Intervention””Divine Intervention - Release group by Slayer””Gold & Platinum - Divine Intervention””Live Intrusion””Undisputed Attitude””Abolish Government/Superficial Love””Release “Slatanic Slaughter: A Tribute to Slayer” by Various Artists””Diabolus in Musica””Soundtrack to the Apocalypse””God Hates Us All””Systematic - Relationships””War at the Warfield””Gold & Platinum - War at the Warfield””Soundtrack to the Apocalypse””Gold & Platinum - Still Reigning””Metallica, Slayer, Iron Mauden Among Winners At Metal Hammer Awards””Eternal Pyre””Eternal Pyre - Slayer release group””Eternal Pyre””Metal Storm Awards 2006””Kerrang! Hall Of Fame””Slayer Wins 'Best Metal' Grammy Award””Slayer Guitarist Jeff Hanneman Dies””Bullet-For My Valentine booed at Metal Hammer Golden Gods Awards””Unholy Aliance””The End Of Slayer?””Slayer: We Could Thrash Out Two More Albums If We're Fast Enough...””'The Unholy Alliance: Chapter III' UK Dates Added”originalet”Megadeth And Slayer To Co-Headline 'Canadian Carnage' Trek”originalet”World Painted Blood””Release “World Painted Blood” by Slayer””Metallica Heading To Cinemas””Slayer, Megadeth To Join Forces For 'European Carnage' Tour - Dec. 18, 2010”originalet”Slayer's Hanneman Contracts Acute Infection; Band To Bring In Guest Guitarist””Cannibal Corpse's Pat O'Brien Will Step In As Slayer's Guest Guitarist”originalet”Slayer’s Jeff Hanneman Dead at 49””Dave Lombardo Says He Made Only $67,000 In 2011 While Touring With Slayer””Slayer: We Do Not Agree With Dave Lombardo's Substance Or Timeline Of Events””Slayer Welcomes Drummer Paul Bostaph Back To The Fold””Slayer Hope to Unveil Never-Before-Heard Jeff Hanneman Material on Next Album””Slayer Debut New Song 'Implode' During Surprise Golden Gods Appearance””Release group Repentless by Slayer””Repentless - Slayer - Credits””Slayer””Metal Storm Awards 2015””Slayer - to release comic book "Repentless #1"””Slayer To Release 'Repentless' 6.66" Vinyl Box Set””BREAKING NEWS: Slayer Announce Farewell Tour””Slayer Recruit Lamb of God, Anthrax, Behemoth + Testament for Final Tour””Slayer lägger ner efter 37 år””Slayer Announces Second North American Leg Of 'Final' Tour””Final World Tour””Slayer Announces Final European Tour With Lamb of God, Anthrax And Obituary””Slayer To Tour Europe With Lamb of God, Anthrax And Obituary””Slayer To Play 'Last French Show Ever' At Next Year's Hellfst””Slayer's Final World Tour Will Extend Into 2019””Death Angel's Rob Cavestany On Slayer's 'Farewell' Tour: 'Some Of Us Could See This Coming'””Testament Has No Plans To Retire Anytime Soon, Says Chuck Billy””Anthrax's Scott Ian On Slayer's 'Farewell' Tour Plans: 'I Was Surprised And I Wasn't Surprised'””Slayer””Slayer's Morbid Schlock””Review/Rock; For Slayer, the Mania Is the Message””Slayer - Biography””Slayer - Reign In Blood”originalet”Dave Lombardo””An exclusive oral history of Slayer”originalet”Exclusive! Interview With Slayer Guitarist Jeff Hanneman”originalet”Thinking Out Loud: Slayer's Kerry King on hair metal, Satan and being polite””Slayer Lyrics””Slayer - Biography””Most influential artists for extreme metal music””Slayer - Reign in Blood””Slayer guitarist Jeff Hanneman dies aged 49””Slatanic Slaughter: A Tribute to Slayer””Gateway to Hell: A Tribute to Slayer””Covered In Blood””Slayer: The Origins of Thrash in San Francisco, CA.””Why They Rule - #6 Slayer”originalet”Guitar World's 100 Greatest Heavy Metal Guitarists Of All Time”originalet”The fans have spoken: Slayer comes out on top in readers' polls”originalet”Tribute to Jeff Hanneman (1964-2013)””Lamb Of God Frontman: We Sound Like A Slayer Rip-Off””BEHEMOTH Frontman Pays Tribute To SLAYER's JEFF HANNEMAN””Slayer, Hatebreed Doing Double Duty On This Year's Ozzfest””System of a Down””Lacuna Coil’s Andrea Ferro Talks Influences, Skateboarding, Band Origins + More””Slayer - Reign in Blood””Into The Lungs of Hell””Slayer rules - en utställning om fans””Slayer and Their Fans Slashed Through a No-Holds-Barred Night at Gas Monkey””Home””Slayer””Gold & Platinum - The Big 4 Live from Sofia, Bulgaria””Exclusive! Interview With Slayer Guitarist Kerry King””2008-02-23: Wiltern, Los Angeles, CA, USA””Slayer's Kerry King To Perform With Megadeth Tonight! - Oct. 21, 2010”originalet”Dave Lombardo - Biography”Slayer Case DismissedArkiveradUltimate Classic Rock: Slayer guitarist Jeff Hanneman dead at 49.”Slayer: "We could never do any thing like Some Kind Of Monster..."””Cannibal Corpse'S Pat O'Brien Will Step In As Slayer'S Guest Guitarist | The Official Slayer Site”originalet”Slayer Wins 'Best Metal' Grammy Award””Slayer Guitarist Jeff Hanneman Dies””Kerrang! Awards 2006 Blog: Kerrang! Hall Of Fame””Kerrang! Awards 2013: Kerrang! Legend”originalet”Metallica, Slayer, Iron Maien Among Winners At Metal Hammer Awards””Metal Hammer Golden Gods Awards””Bullet For My Valentine Booed At Metal Hammer Golden Gods Awards””Metal Storm Awards 2006””Metal Storm Awards 2015””Slayer's Concert History””Slayer - Relationships””Slayer - Releases”Slayers officiella webbplatsSlayer på MusicBrainzOfficiell webbplatsSlayerSlayerr1373445760000 0001 1540 47353068615-5086262726cb13906545x(data)6033143kn20030215029