diff --git a/README.md b/README.md index 9e08685..9e8c8ea 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,75 @@ java-generator-functions ======================== -An implementation of Python-like generator functions in Java. This repository contains a single class, `Generator` with a method `yield(...)` which can be used to mimic the behaviour of the `yield` keyword in Python. +An implementation of Python-like generator functions in Java. This repository contains a functional interface, `GeneratorFunc`, which accepts an object with a method, `yield(...)`, that can be used to mimic the behaviour of the `yield` keyword in Python. Examples -------- The following is a simple generator that yields `1` and then `2`: - Generator simpleGenerator = new Generator() { - public void run() throws InterruptedException { - yield(1); - // Some logic here... - yield(2); - } - }; - for (Integer element : simpleGenerator) - System.out.println(element); - // Prints "1", then "2". +```java +GeneratorFunc simpleGenerator = s -> { + s.yield(1); + // Some logic here... + s.yield(2); +}; + +for (Integer element : simpleGenerator) + System.out.println(element); +// Prints "1", then "2". +``` Infinite generators are also possible: - Generator infiniteGenerator = new Generator() { - public void run() throws InterruptedException { - while (true) - yield(1); - } - }; +```java +GeneratorFunc infiniteGenerator = s -> { + while (true) + s.yield(1); +}; +``` + +You can even use a generator to create a (parallel) `Stream`: + +```java +GeneratorFunc infiniteGenerator = s -> { + int i = 0; + while (true) { + s.yield(i++); + } +}; + +infiniteGenerator.stream().limit(100).parallel() // and so on +``` + +Or, equivalently: + +```java +// Note that the generic parameter is necessary, or else Java can't determine +// the generator's type. +Generator.stream(s -> { + int i = 0; + while (true) { + s.yield(i++); + } +}).limit(100).parallel() // and so on +``` + +If you need to use an anonymous inner class, it is more concise to have it extend `Generator`, at the cost of losing statelessness: + +```java +Generator infiniteGenerator = new Generator() { + public void run() throws InterruptedException { + while (true) + yield(1); + } +}; +``` + +You can iterate over a generator multiple times, resulting in multiple calls to the lambda or `run` method. If the generator modifies some state, you can expect that state to be modified each time you iterate over the generator (or create a `Stream` from it). -The `Generator` class lies in package `io.herrmann.generator`. So you need to `import io.herrmann.generator.Generator;` in order for the above examples to work. +For more examples, see [GeneratorTest.java](src/test/java/io/herrmann/generator/GeneratorTest.java). + +The `Generator` class and `GeneratorFunc` interface lie in the package `io.herrmann.generator`, so you need to `import io.herrmann.generator.*;` in order for the above examples to work. Usage ----- @@ -38,37 +80,43 @@ This package is hosted as a Maven repository with the following url: To use it from Maven, add the following to your `pom.xml`: - - ... - - ... - - java-generator-functions - http://dl.bintray.com/filipmalczak/maven - - +```xml + + ... + ... - - - io.herrmann - java-generator-functions - 1.0 - - - + + java-generator-functions + http://dl.bintray.com/filipmalczak/maven + + + ... + + + io.herrmann + java-generator-functions + 1.0 + + + +``` For Gradle: - compile(group: 'io.herrmann', name: 'java-generator-functions', version: '1.0') +```gradle +compile(group: 'io.herrmann', name: 'java-generator-functions', version: '1.0') +``` Caveats and Performance ----------------------- -The `Generator` class internally works with a Thread to produce the items. It does ensure that no Threads stay around if the corresponding Generator is no longer used. However: +The `Generator` library internally works with a Thread to produce the items. It does ensure that no Threads stay around if the corresponding Generator is no longer used. However: **If too many `Generator`s are created before the JVM gets a chance to garbage collect the old ones, you may encounter `OutOfMemoryError`s. This problem most strongly presents itself on OS X where the maximum number of Threads is significantly lower than on other OSs (around 2000).** -The performance is obviously not great but not too shabby either. On my machine with a dual core i5 CPU @ 2.67 GHz, 1000 items can be produced in < 0.03s. +The performance is obviously not great, but not too shabby either. On my machine with a dual core i5 CPU @ 2.67 GHz, 1000 items can be produced in < 0.03s. + +This version requires Java 8, as it takes advantage of functional interfaces in its API and provides integration with the Streams API. If you need support for an older version of Java, use version 1.0 of this library. Contributing ------------ -Contributions and pull requests are welcome. Please ensure that `mvn test` still passes and add any unit tests as you see fit. Please also follow the same coding conventions, in particular the line limit of 80 characters and the use of tabs instead of spaces. \ No newline at end of file +Contributions and pull requests are welcome. Please ensure that `mvn test` still passes and add any unit tests as you see fit. Please also follow the same coding conventions, in particular the line limit of 80 characters and the use of tabs instead of spaces. diff --git a/pom.xml b/pom.xml index 159809e..27c000f 100644 --- a/pom.xml +++ b/pom.xml @@ -44,4 +44,19 @@ test - \ No newline at end of file + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.5.1 + + 1.8 + 1.8 + + + + + + diff --git a/src/main/java/io/herrmann/generator/Generator.java b/src/main/java/io/herrmann/generator/Generator.java index 78ffccd..70b3f59 100644 --- a/src/main/java/io/herrmann/generator/Generator.java +++ b/src/main/java/io/herrmann/generator/Generator.java @@ -1,124 +1,49 @@ package io.herrmann.generator; import java.util.Iterator; -import java.util.NoSuchElementException; +import java.util.stream.Stream; +import java.util.stream.StreamSupport; /** - * This class allows specifying Python generator-like sequences. For examples, - * see the JUnit test case. - * - * The implementation uses a separate Thread to produce the sequence items. This - * is certainly not as fast as eg. a for-loop, but not horribly slow either. On - * a machine with a dual core i5 CPU @ 2.67 GHz, 1000 items can be produced in - * < 0.03s. - * - * By overriding finalize(), the class takes care not to leave any Threads - * running longer than necessary. + * Implementation of {@link GeneratorFunc} as an abstract class. This class + * mainly exists for backwards compatibility, but it can also cut down some of + * the boilerplate when using an anonymous inner class instead of a lambda. + * However, unlike a {@link GeneratorFunc}, this class is not stateless, and + * cannot be used concurrently. */ -public abstract class Generator implements Iterable { +public abstract class Generator implements GeneratorFunc { - private class Condition { - private boolean isSet; - public synchronized void set() { - isSet = true; - notify(); - } - public synchronized void await() throws InterruptedException { - try { - if (isSet) - return; - wait(); - } finally { - isSet = false; - } - } - } - - static ThreadGroup THREAD_GROUP; - - Thread producer; - private boolean hasFinished; - private final Condition itemAvailableOrHasFinished = new Condition(); - private final Condition itemRequested = new Condition(); - private T nextItem; - private boolean nextItemAvailable; - private RuntimeException exceptionRaisedByProducer; + private GeneratorIterator iter; @Override - public Iterator iterator() { - return new Iterator() { - @Override - public boolean hasNext() { - return waitForNext(); - } - @Override - public T next() { - if (!waitForNext()) - throw new NoSuchElementException(); - nextItemAvailable = false; - return nextItem; - } - @Override - public void remove() { - throw new UnsupportedOperationException(); - } - private boolean waitForNext() { - if (nextItemAvailable) - return true; - if (hasFinished) - return false; - if (producer == null) - startProducer(); - itemRequested.set(); - try { - itemAvailableOrHasFinished.await(); - } catch (InterruptedException e) { - hasFinished = true; - } - if (exceptionRaisedByProducer != null) - throw exceptionRaisedByProducer; - return !hasFinished; - } - }; + public void run(GeneratorIterator gen) throws InterruptedException { + run(); } protected abstract void run() throws InterruptedException; protected void yield(T element) throws InterruptedException { - nextItem = element; - nextItemAvailable = true; - itemAvailableOrHasFinished.set(); - itemRequested.await(); + iter.yield(element); } - private void startProducer() { - assert producer == null; - if (THREAD_GROUP == null) - THREAD_GROUP = new ThreadGroup("generatorfunctions"); - producer = new Thread(THREAD_GROUP, new Runnable() { - @Override - public void run() { - try { - itemRequested.await(); - Generator.this.run(); - } catch (InterruptedException e) { - // No need to do anything here; Remaining steps in run() - // will cleanly shut down the thread. - } catch (RuntimeException e) { - exceptionRaisedByProducer = e; - } - hasFinished = true; - itemAvailableOrHasFinished.set(); - } - }); - producer.setDaemon(true); - producer.start(); + @Override + public Iterator iterator() { + iter = new GeneratorIterator<>(this); + return iter; } - @Override - protected void finalize() throws Throwable { - producer.interrupt(); - producer.join(); - super.finalize(); + /** + * Creates a {@link Stream} from a {@link GeneratorFunc}. For cases where + * the generator isn't a lambda passed directly, the instance method {@link + * #stream()} is generally more concise. + * + * @param g The generator + * @return An ordered, sequential (non-parallel) stream of elements yielded + * by the generator + * @see #stream() + */ + public static Stream stream(GeneratorFunc g) { + return StreamSupport.stream(g.spliterator(), false); } + } diff --git a/src/main/java/io/herrmann/generator/GeneratorFunc.java b/src/main/java/io/herrmann/generator/GeneratorFunc.java new file mode 100644 index 0000000..65770fd --- /dev/null +++ b/src/main/java/io/herrmann/generator/GeneratorFunc.java @@ -0,0 +1,55 @@ +package io.herrmann.generator; + +import java.util.Iterator; +import java.util.Spliterator; +import java.util.Spliterators; +import java.util.stream.Stream; + +/** + * This functional interface allows specifying Python generator-like sequences. + * For examples, see the JUnit test case. + * + * The implementation uses a separate Thread to produce the sequence items. This + * is certainly not as fast as eg. a for-loop, but not horribly slow either. On + * a machine with a dual core i5 CPU @ 2.67 GHz, 1000 items can be produced in + * < 0.03s. + * + * By overriding finalize(), the underlying iterator takes care not to leave any + * Threads running longer than necessary. + * + * @see Generator + */ +@FunctionalInterface +public interface GeneratorFunc extends Iterable { + + @Override + public default Iterator iterator() { + return new GeneratorIterator<>(this); + } + + public void run(GeneratorIterator gen) throws InterruptedException; + + /** + * Returns an ordered {@link Spliterator} consisting of elements yielded by + * this {@link GeneratorFunc}. + */ + @Override + default Spliterator spliterator() { + return Spliterators.spliteratorUnknownSize(iterator(), + Spliterator.ORDERED); + } + + /** + * Creates a {@link Stream} from a {@link GeneratorFunc}. If you are trying + * to call this on a lambda, you should either use the static method {@link + * Generator#stream()} or assign it to a variable first. + * + * @param g The generator + * @return An ordered, sequential (non-parallel) stream of elements yielded + * by the generator + */ + public default Stream stream() { + return Generator.stream(this); + } + +} diff --git a/src/main/java/io/herrmann/generator/GeneratorIterator.java b/src/main/java/io/herrmann/generator/GeneratorIterator.java new file mode 100644 index 0000000..5a5f2b8 --- /dev/null +++ b/src/main/java/io/herrmann/generator/GeneratorIterator.java @@ -0,0 +1,124 @@ +package io.herrmann.generator; + +import java.util.Iterator; +import java.util.NoSuchElementException; +import java.util.Objects; + +/** + * This is the class that contains most of the logic for the {@link Generator} + * and {@link GeneratorFunc} classes. It is possible to create several of these + * for one {@link GeneratorFunc}, but use with caution if the generator function + * is not stateless. + */ +public final class GeneratorIterator implements Iterator { + + static ThreadGroup THREAD_GROUP; + + GeneratorFunc gen; + + public GeneratorIterator(GeneratorFunc gen) { + Objects.requireNonNull(gen); + this.gen = gen; + } + + @Override + public boolean hasNext() { + return waitForNext(); + } + + @Override + public T next() { + if (!waitForNext()) + throw new NoSuchElementException(); + nextItemAvailable = false; + return nextItem; + } + + @Override + public void remove() { + throw new UnsupportedOperationException(); + } + + private boolean waitForNext() { + if (nextItemAvailable) + return true; + if (hasFinished) + return false; + if (producer == null) + startProducer(); + itemRequested.set(); + try { + itemAvailableOrHasFinished.await(); + } catch (InterruptedException e) { + hasFinished = true; + } + if (exceptionRaisedByProducer != null) + throw exceptionRaisedByProducer; + return !hasFinished; + } + + private class Condition { + private boolean isSet; + public synchronized void set() { + isSet = true; + notify(); + } + public synchronized void await() throws InterruptedException { + try { + if (isSet) + return; + wait(); + } finally { + isSet = false; + } + } + } + + Thread producer; + private boolean hasFinished; + private final Condition itemAvailableOrHasFinished = new Condition(); + private final Condition itemRequested = new Condition(); + private T nextItem; + private boolean nextItemAvailable; + private RuntimeException exceptionRaisedByProducer; + + public void yield(T element) throws InterruptedException { + nextItem = element; + nextItemAvailable = true; + itemAvailableOrHasFinished.set(); + itemRequested.await(); + } + + private void startProducer() { + if (producer != null) { + throw new IllegalStateException( + "Can't use the same GeneratorIterator twice!"); + } + + if (THREAD_GROUP == null) + THREAD_GROUP = new ThreadGroup("generatorfunctions"); + producer = new Thread(THREAD_GROUP, () -> { + try { + itemRequested.await(); + gen.run(this); + } catch (InterruptedException e) { + // No need to do anything here; Remaining steps in run() + // will cleanly shut down the thread. + } catch (RuntimeException e) { + exceptionRaisedByProducer = e; + } + hasFinished = true; + itemAvailableOrHasFinished.set(); + }); + producer.setDaemon(true); + producer.start(); + } + + @Override + protected void finalize() throws Throwable { + producer.interrupt(); + producer.join(); + super.finalize(); + } + +} \ No newline at end of file diff --git a/src/test/java/io/herrmann/generator/GeneratorTest.java b/src/test/java/io/herrmann/generator/GeneratorTest.java index 810c424..6ae84b2 100644 --- a/src/test/java/io/herrmann/generator/GeneratorTest.java +++ b/src/test/java/io/herrmann/generator/GeneratorTest.java @@ -2,99 +2,204 @@ import org.junit.Test; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Iterator; -import java.util.List; +import java.awt.Point; +import java.awt.Rectangle; +import java.util.*; +import java.util.stream.*; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; public class GeneratorTest { + + private static GeneratorFunc emptyGenerator = s -> {}; + + private static GeneratorFunc infiniteGenerator = s -> { + while (true) { + s.yield(1); + } + }; + @Test public void testEmptyGenerator() { - assertEquals(new ArrayList(), list(new EmptyGenerator())); - } - private class EmptyGenerator extends Generator { - @Override - protected void run() { - } + assertEquals(new ArrayList(), list(emptyGenerator)); } + public static List list(Iterable iterable) { List result = new ArrayList(); - for (T item : iterable) - result.add(item); + iterable.forEach(result::add); return result; } + @Test public void testOneEltGenerator() { List oneEltList = Arrays.asList(1); assertEquals(oneEltList, list(new ListGenerator(oneEltList))); } + private class ListGenerator extends Generator { private final List elements; public ListGenerator(List elements) { this.elements = elements; } - protected void run() throws InterruptedException { + + public void run() throws InterruptedException { for (T element : elements) yield(element); } } + @Test public void testTwoEltGenerator() { List twoEltList = Arrays.asList(1, 2); - assertEquals(twoEltList, list(new ListGenerator(twoEltList))); + assertEquals(twoEltList, list(new ListGenerator<>(twoEltList))); } + @Test public void testInfiniteGenerator() { - InfiniteGenerator generator = new InfiniteGenerator(); - testInfiniteGenerator(generator); + GeneratorFunc generator = infiniteGenerator; + testInfiniteGenerator(generator.iterator()); } - public void testInfiniteGenerator(InfiniteGenerator generator) { + + public void testInfiniteGenerator(Iterator generatorIterator) { int NUM_ELTS_TO_INSPECT = 1000; - Iterator generatorIterator = generator.iterator(); for (int i=0; i < NUM_ELTS_TO_INSPECT; i++) { assertTrue(generatorIterator.hasNext()); assertEquals(1, (int) generatorIterator.next()); } } - private class InfiniteGenerator extends Generator { - @Override - protected void run() throws InterruptedException { - while (true) - yield(1); - } - } + @Test public void testInfiniteGeneratorLeavesNoRunningThreads() throws Throwable { - InfiniteGenerator generator = new InfiniteGenerator(); - testInfiniteGenerator(generator); - generator.finalize(); - assertEquals(Thread.State.TERMINATED, generator.producer.getState()); + GeneratorFunc generator = infiniteGenerator; + GeneratorIterator iterator = + (GeneratorIterator) generator.iterator(); + testInfiniteGenerator(iterator); + iterator.finalize(); + assertEquals(Thread.State.TERMINATED, + iterator.producer.getState()); } + @SuppressWarnings("serial") private class CustomRuntimeException extends RuntimeException {} - private class GeneratorRaisingException extends Generator { - @Override - protected void run() throws InterruptedException { - throw new CustomRuntimeException(); - } - } - @Test(expected = CustomRuntimeException.class) public void testGeneratorRaisingExceptionHasNext() { - GeneratorRaisingException generator = new GeneratorRaisingException(); + GeneratorFunc generator = s -> { throw new CustomRuntimeException(); }; Iterator iterator = generator.iterator(); iterator.hasNext(); } @Test(expected = CustomRuntimeException.class) public void testGeneratorRaisingExceptionNext() { - GeneratorRaisingException generator = new GeneratorRaisingException(); + GeneratorFunc generator = s -> { throw new CustomRuntimeException(); }; Iterator iterator = generator.iterator(); iterator.next(); } -} \ No newline at end of file + @Test + public void testUseInStream() { + List nums = Arrays.asList(0, 1, 2, 3, 4, 5); + + // Note that the generic parameter is necessary, or else Java can't + // determine the generator's type. + int sum = Generator.stream(s -> { + for (int n : nums) { + s.yield(n); + } + }).limit(nums.size()).mapToInt(x -> x).sum(); + + assertEquals(sum, nums.stream().mapToInt(x -> x).sum()); + } + + @Test + public void testUseInParallelStream() { + // A slightly more realistic usage example: generate a set of lattice + // points in a given rectangle + Rectangle r = new Rectangle(2, 3, 2, 4); + + GeneratorFunc pointGen = s -> { + for (int x = 0; x < 10; x++) { + for (int y = 0; y < 10; y ++) { + s.yield(new Point(x, y)); + } + } + }; + + Set ps = pointGen.stream().parallel() + .filter(r::contains) + .collect(Collectors.toSet()); + + // For comparison, here's what you might have to do to get a parallel + // stream without this library. More concise? Yes. More + // intuitive/readable? Probably not. + Set ps1 = IntStream.range(0, 100) + .mapToObj(i -> new Point(i % 10, i / 10)) + .parallel() + .filter(r::contains) + .collect(Collectors.toSet()); + + // Generate it the old fashioned way for comparison + Set ps2 = new HashSet<>(); + for (int x = 0; x < 10; x++) { + for (int y = 0; y < 10; y ++) { + if (r.contains(x, y)) { + ps2.add(new Point(x, y)); + } + } + } + + assertEquals(ps2, ps); + assertEquals(ps2, ps1); + } + + @Test + public void testGeneratorClassUseInParallelStream() { + // An infinite generator for fibonacci numbers! + Generator fibs = new Generator() { + @Override + protected void run() throws InterruptedException { + int a = 0, b = 1; + while (true) { + yield(a); + int next = a + b; + a = b; + b = next; + } + } + }; + + int sum1 = fibs.stream().limit(45).mapToInt(x -> x).sum(); + assertEquals(1836311902, sum1); + + // An old-fashioned Stream of fibonacci numbers + int sum2 = Stream.iterate(new int[]{ 0, 1 }, + c -> new int[]{ c[1], c[0] + c[1] }) + .limit(45).parallel().mapToInt(a -> a[0]).sum(); + assertEquals(1836311902, sum2); + } + + @Test + public void testReuseGeneratorClass() { + Generator naturalNumbers = new Generator() { + @Override + protected void run() throws InterruptedException { + int i = 0; + while (true) { + yield(i++); + } + } + }; + + // sum of 0-4 + int sum = naturalNumbers.stream().limit(5).mapToInt(x -> x).sum(); + + assertEquals(sum, 10); + + // sum of 0-9 + sum = naturalNumbers.stream() .limit(10).mapToInt(x -> x).sum(); + + assertEquals(sum, 45); + } + +}