cascading
Class JoinFieldedPipesPlatformTest

java.lang.Object
  extended by junit.framework.Assert
      extended by junit.framework.TestCase
          extended by cascading.CascadingTestCase
              extended by cascading.PlatformTestCase
                  extended by cascading.JoinFieldedPipesPlatformTest
All Implemented Interfaces:
Serializable, junit.framework.Test

public class JoinFieldedPipesPlatformTest
extends PlatformTestCase

See Also:
Serialized Form

Nested Class Summary
static class JoinFieldedPipesPlatformTest.AllComparator
           
 
Field Summary
 
Fields inherited from class cascading.PlatformTestCase
name, ROOT_OUTPUT_PATH
 
Constructor Summary
JoinFieldedPipesPlatformTest()
           
 
Method Summary
 void testCross()
           
 void testJoin()
           
 void testJoinAfterEvery()
          Method testCoGroupAfterEvery tests that a tmp tap is inserted after the Every in the cogroup join
 void testJoinAroundJoinLeftMost()
          Same source as leftmost
 void testJoinAroundJoinRightMost()
          Same source as rightmost

should be a single job as the same file accumulates into the joins

 void testJoinAroundJoinRightMostSwapped()
          Upper as leftmost and rightmost forcing two jobs
 void testJoinDiffFields()
           
 void testJoinFilteredBranch()
          this test intentionally filters out all values so the intermediate tap is empty.
 void testJoinGroupBy()
           
 void testJoinGroupByJoin()
           
 void testJoinInner()
          1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null h1 null H1

 void testJoinInnerOuter()
          1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1

 void testJoinInnerSingleField()
           
 void testJoinMergeGroupBy()
          When run against a cluster a Merge before a GroupBy can hide the streamed/accumulated nature of a branch.
 void testJoinMixed()
          1 a 5 b 6 c 5 b 5 e

1 A 2 B 3 C 4 D 5 E

1 a 2 b 3 c 4 d 5 e

1 a 1 A 1 a - - 2 B 2 b - - 3 C 3 c - - 4 D 4 d 5 b 5 E 5 e 5 e 5 E 5 e

 void testJoinNone()
           
 void testJoinOuter()
          /** 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null null null 6 F1 null null 6 F2 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1

 void testJoinOuterInner()
          1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null null 6 F1 null null 6 F2 null h1 null H1

 void testJoinSamePipe()
           
 void testJoinSamePipe2()
           
 void testJoinSamePipe3()
           
 void testJoinSamePipeAroundGroupBy()
           
 void testJoinSamePipeName()
           
 void testJoinSameSourceIntoJoin()
          here the same file is fed into the same HashJoin.
 void testJoinSelf()
           
 void testJoinsIntoCoGroup()
           
 void testJoinsIntoCoGroupLhs()
          This test results in two MR jobs because one join feeds into the accumulated side of the second.
 void testJoinsIntoCoGroupLhsSwappedJoin()
          This test results in one MR jobs because one join feeds into the streamed side of the second.
 void testJoinsIntoCoGroupRhs()
           
 void testJoinsIntoGroupBy()
          Tests that two independent streamed sources with loadable tributaries properly plan into a GroupBy without loading unused sources
 void testJoinWithHasher()
          Tests Hasher being honored even if default comparator is null.
 void testJoinWithUnknowns()
           
 
Methods inherited from class cascading.PlatformTestCase
copyFromLocal, disableDebug, getOutputPath, getPathElements, getPlatform, getPlatformName, getProperties, getRootPath, getTestCaseName, getTestName, getTestRoot, installPlatform, setUp, tearDown
 
Methods inherited from class cascading.CascadingTestCase
asCollection, asCollection, asCollection, asCollection, asList, asList, asSet, asSet, getSinkAsList, getSourceAsList, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength
 
Methods inherited from class junit.framework.TestCase
assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertTrue, assertTrue, countTestCases, createResult, fail, fail, failNotEquals, failNotSame, failSame, format, getName, run, run, runBare, runTest, setName, toString
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

JoinFieldedPipesPlatformTest

public JoinFieldedPipesPlatformTest()
Method Detail

testCross

public void testCross()
               throws Exception
Throws:
Exception

testJoin

public void testJoin()
              throws Exception
Throws:
Exception

testJoinSamePipeName

public void testJoinSamePipeName()
                          throws Exception
Throws:
Exception

testJoinWithUnknowns

public void testJoinWithUnknowns()
                          throws Exception
Throws:
Exception

testJoinFilteredBranch

public void testJoinFilteredBranch()
                            throws Exception
this test intentionally filters out all values so the intermediate tap is empty. this tap is cogrouped with a new stream using an outerjoin.

Throws:
Exception

testJoinSelf

public void testJoinSelf()
                  throws Exception
Throws:
Exception

testJoinAfterEvery

public void testJoinAfterEvery()
                        throws Exception
Method testCoGroupAfterEvery tests that a tmp tap is inserted after the Every in the cogroup join

Throws:
Exception - when

testJoinInnerSingleField

public void testJoinInnerSingleField()
                              throws Exception
Throws:
Exception

testJoinInner

public void testJoinInner()
                   throws Exception
1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null h1 null H1

Throws:
Exception

testJoinOuter

public void testJoinOuter()
                   throws Exception
/** 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null null null 6 F1 null null 6 F2 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1

Throws:
Exception

testJoinInnerOuter

public void testJoinInnerOuter()
                        throws Exception
1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1

Throws:
Exception

testJoinOuterInner

public void testJoinOuterInner()
                        throws Exception
1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null null 6 F1 null null 6 F2 null h1 null H1

Throws:
Exception

testJoinMixed

public void testJoinMixed()
                   throws Exception
1 a 5 b 6 c 5 b 5 e

1 A 2 B 3 C 4 D 5 E

1 a 2 b 3 c 4 d 5 e

1 a 1 A 1 a - - 2 B 2 b - - 3 C 3 c - - 4 D 4 d 5 b 5 E 5 e 5 e 5 E 5 e

Throws:
Exception

testJoinDiffFields

public void testJoinDiffFields()
                        throws Exception
Throws:
Exception

testJoinGroupBy

public void testJoinGroupBy()
                     throws Exception
Throws:
Exception

testJoinSamePipe

public void testJoinSamePipe()
                      throws Exception
Throws:
Exception

testJoinSamePipe2

public void testJoinSamePipe2()
                       throws Exception
Throws:
Exception

testJoinSamePipe3

public void testJoinSamePipe3()
                       throws Exception
Throws:
Exception

testJoinAroundJoinRightMost

public void testJoinAroundJoinRightMost()
                                 throws Exception
Same source as rightmost

should be a single job as the same file accumulates into the joins

Throws:
Exception

testJoinAroundJoinLeftMost

public void testJoinAroundJoinLeftMost()
                                throws Exception
Same source as leftmost

Throws:
Exception

testJoinAroundJoinRightMostSwapped

public void testJoinAroundJoinRightMostSwapped()
                                        throws Exception
Upper as leftmost and rightmost forcing two jobs

Throws:
Exception

testJoinGroupByJoin

public void testJoinGroupByJoin()
                         throws Exception
Throws:
Exception

testJoinSameSourceIntoJoin

public void testJoinSameSourceIntoJoin()
                                throws Exception
here the same file is fed into the same HashJoin.

This is three jobs.

a temp tap is inserted before the accumulated branch for two reasons on the common HashJoin

it is assumed the accumulated side is filtered down, so pushing to disk will preserve io if accumulated side was streamed instead via a fork, only part of the file will accumulate into the HashJoin

/-T-\ <-- accumulated T HJ \---/ <-- streamed

Throws:
Exception

testJoinsIntoGroupBy

public void testJoinsIntoGroupBy()
                          throws Exception
Tests that two independent streamed sources with loadable tributaries properly plan into a GroupBy without loading unused sources

Throws:
Exception

testJoinSamePipeAroundGroupBy

public void testJoinSamePipeAroundGroupBy()
                                   throws Exception
Throws:
Exception

testJoinsIntoCoGroupLhs

public void testJoinsIntoCoGroupLhs()
                             throws Exception
This test results in two MR jobs because one join feeds into the accumulated side of the second. A mapper can only stream on branch at a time forcing a temp file between the mappers. see next test for swapped join

Throws:
Exception

testJoinsIntoCoGroupLhsSwappedJoin

public void testJoinsIntoCoGroupLhsSwappedJoin()
                                        throws Exception
This test results in one MR jobs because one join feeds into the streamed side of the second.

Throws:
Exception

testJoinsIntoCoGroupRhs

public void testJoinsIntoCoGroupRhs()
                             throws Exception
Throws:
Exception

testJoinsIntoCoGroup

public void testJoinsIntoCoGroup()
                          throws Exception
Throws:
Exception

testJoinWithHasher

public void testJoinWithHasher()
                        throws Exception
Tests Hasher being honored even if default comparator is null.

Throws:
Exception

testJoinNone

public void testJoinNone()
                  throws Exception
Throws:
Exception

testJoinMergeGroupBy

public void testJoinMergeGroupBy()
                          throws Exception
When run against a cluster a Merge before a GroupBy can hide the streamed/accumulated nature of a branch. commented code is for troubleshooting.

Throws:
Exception


Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.