Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
181 commits
Select commit Hold shift + click to select a range
3e7354d
Removed dead code.
Sep 23, 2014
bca4c8f
More code cleanups.
Sep 23, 2014
2fec59b
Setting up the structure.
Dec 28, 2014
16c5d07
Ignore built files.
Dec 28, 2014
0953c9f
Second part of the moving.
Dec 28, 2014
e34d776
Moving another directory.
Dec 28, 2014
0bf0a90
Moved most of plugins.Library.util.concurrent.
Dec 28, 2014
561424f
Moved the rest of plugin.Library.util.concurrent.
Dec 28, 2014
32daec1
Moved first (and simple) part of the plugins.Library.util.
Dec 28, 2014
c3c35a3
Moved two more classes from plugins.Library.util.
Dec 28, 2014
3a024be
Moved some serialisers from plugins.Library.io.serial.
Dec 28, 2014
590818d
Cleanups to allow moving of the classes.
Dec 28, 2014
8077166
Moved the rest of the classes from plugins.Library.util.
Dec 28, 2014
bed5daa
Removed logging to allow for the move.
Dec 28, 2014
3e0d321
Moved almost all files from plugins.Library.io and .serial.
Dec 28, 2014
fa6053e
Refactoring the yaml-code to factor out the Freenet classes.
Dec 28, 2014
9d6c6ed
Moved the reader/writer and entries.
Dec 28, 2014
76954de
Re-enabled compilation using ant.
Jan 2, 2015
4cb7b8e
Fixes of first problem found when running on old index.
Jan 3, 2015
147c28e
Refactored to simplify code and include object reading time in the lo…
Jan 3, 2015
83f2efd
Added more stack traces in the log to be able to spot complex problems.
Jan 3, 2015
e035a8e
Reverted the type changes in the TermPageEntry to allow old indexes t…
Jan 3, 2015
f540a0b
Some eclipse settings for uploader.
Jan 3, 2015
64610ff
Some small simple refactorings to prepare moving ProtoIndex to shared…
Jan 3, 2015
bd7f64a
Changed FreenetURI to String in index/Proto*.java.
Jan 3, 2015
8c62513
Moved Index.java interface and URIEntry, URIKey to the shared.
Jan 3, 2015
ab72b94
Created a register for the factory for LiveArchiver.
Jan 3, 2015
a11bb25
Moved the ProtoIndex to shared.
Jan 3, 2015
65f9ae2
First framework of the uploader compiling.
Jan 4, 2015
95f21f3
Create the uploader jar.
Jan 4, 2015
0e29164
Reordered the removelistener to reduce concurrentexceptions.
Jan 4, 2015
df0e4e0
Added spider compatibility when reading the terms.
Jan 4, 2015
3c485c8
Improved the proof of concept.
Jan 4, 2015
f861df0
Build more packages.
Jan 25, 2015
405c482
Some more work, slowly approaching uploading.
Feb 14, 2015
b7c47ca
New workspace with project named a little differently.
Mar 15, 2015
5f5e1a1
Added debug trace.
Mar 16, 2015
a6635ed
Abandoned the idea of separating the Archiver and joined together to
Mar 20, 2015
1e8a345
Added upload to the fcp-adapter.
Apr 5, 2015
57cd5f0
Removed tabs.
Apr 6, 2015
1c39122
Use the file the second time, to avoid locking up the writer.
Apr 6, 2015
452d905
Second attempt at removing the tabs.
Apr 6, 2015
2b7e0e1
Moved the handling of the close-detection to get the right close mess…
Apr 8, 2015
f60365a
Created the processing of files and creating the on file index.
Apr 9, 2015
8450a13
Let also this pool go down to 0 threads when idle so that the script …
Apr 10, 2015
dafbfcf
Tune the parameters for amount of stored entries and amount of files.
Apr 10, 2015
e2071fd
Don't finish the operation until the PutSuccessful is received.
Apr 10, 2015
1f370c3
First working implementation of the IndexPeeker to group Terms.
Apr 10, 2015
b46dfb2
Refactored the interface to the IndexPeeker.
Apr 10, 2015
d0a3190
Removed not used code and imports.
Apr 10, 2015
744bcf0
Removed not used field.
Apr 11, 2015
e46a7e1
Implementation that does the actual peeking in the Index to select on…
Apr 11, 2015
e5f699c
Made the directory creator resilient against already existing directo…
Apr 11, 2015
fc0050e
Removed confusing trace message.
Apr 11, 2015
c8f1ffd
Let the files be a little bigger. (1000000 entries ~ 200M).
Apr 15, 2015
bf349a1
Buggfix: delay deflating the tree until it is all filled.
Apr 15, 2015
f620113
Refactored IndexPeeker to make it easy to do more section at the same…
Apr 15, 2015
380f66f
Factored out the setting up and taking down of the Fcp session.
Apr 16, 2015
4053787
Buggfix for the non-session case.
Apr 16, 2015
195c8df
Changed the handling of files to process all not previously terms eac…
Apr 17, 2015
9f9b527
Made it possible to specify the client name to FcpSession.
Apr 24, 2015
ede1ff6
Process the files in the correct order.
Apr 25, 2015
e292cce
Factored out the FileClientPutter.
May 5, 2015
60731c0
Moved the cleanup thread entirely into the FileClientPutter.
May 6, 2015
94cf037
Reverted back to 1148 since it didn't work.
May 6, 2015
36f4017
Initial revision of an index downloader.
May 24, 2015
5298395
Improved code quality.
Jun 3, 2015
5809473
New calculation of estimate of entire tree.
Jun 3, 2015
b1d5a91
Added function to include the contents of several selected file, if s…
Jun 3, 2015
0aa5f83
Adjustments in the calculation of how many selected to include.
Jun 6, 2015
c9441df
Calculate how many uploads that are resurrected.
Jun 6, 2015
d349b19
The new known good getter changed name in FCPlib.
Jun 8, 2015
e7c183e
Attempted to tune to a better approximation of the size of an upload.
Jun 8, 2015
1efc22a
Changed the eclipse project names and location of the fcp lib.
Jun 8, 2015
7988d48
Updated the Eclipse project name.
debbiedub Jun 11, 2015
30f9bef
Slow down the build up of the queue of uploads.
debbiedub Jun 11, 2015
e5904b2
Tuned the size of the uploads down a bit.
debbiedub Jun 12, 2015
7f1c88e
Reduce ant warnings.
debbiedub Jul 8, 2015
0bda01e
Moved the index selection to a separate web page.
debbiedub Jul 9, 2015
6032d07
Force my testing index into the mix.
debbiedub Jun 16, 2015
a91d247
Add retrieval of index of sites for a word.
debbiedub Jun 23, 2015
c531024
Reduced logging when downloading.
debbiedub Jun 23, 2015
575095c
Script to run the upload perpetually.
debbiedub Jul 12, 2015
96a24ea
Corrected the logic for adding multiple selected in an upload.
debbiedub Jul 12, 2015
554914a
Take down the uploader jobs service.
debbiedub Sep 24, 2015
873e9bc
Add log messages when URIs are not fetched.
debbiedub Sep 27, 2015
2e44d79
Moved the pruning of the trees to avoid always having at least two of…
debbiedub Sep 27, 2015
7e659c7
End the waiting indefinately.
debbiedub Dec 6, 2015
0a8ac89
Saving the unpacked object for examining it in the debugger.
debbiedub Dec 7, 2015
7115936
Fixed problem with specific downloads. Added more logging.
debbiedub Feb 20, 2016
b5f214c
Accept and count that the CHK:s has occasionally changed.
debbiedub Apr 16, 2016
06d0b39
Adjusted the logging.
debbiedub Apr 27, 2016
551672f
Reduce the amount of downloading while there is uploading.
debbiedub May 1, 2016
b9191b8
Added max queue-len.
debbiedub May 4, 2016
d134cb4
Count and log failed uploads.
debbiedub May 14, 2016
b7e5966
Factored out the parsing of the contents of the files.
debbiedub May 14, 2016
d128eb4
Change back to 10 simultaneous fetches. 30 doesn't seem quicker.
debbiedub May 15, 2016
267368c
Added a function to move all files connected from a CHK.
debbiedub May 15, 2016
2a94540
Fixed some exceptions. Adjusted logging.
debbiedub Jun 10, 2016
8bd931c
Factored out aging of running fetches.
debbiedub Aug 16, 2016
1cfdf5b
Clean the title from not-printable characters.
debbiedub Aug 24, 2016
d3c14b7
Comment to make it easer to run the move job.
debbiedub Aug 24, 2016
fd2eb8c
Removed callback message.
debbiedub Nov 5, 2016
a17c1af
Added delay.
debbiedub Jul 2, 2017
6dec3a8
Added yaml sources location.
debbiedub Jul 2, 2017
3672757
Partial work done to the restore the FreenetURI...
debbiedub Dec 19, 2015
faa221d
Fixes to get rid of Eclipse warnings.
debbiedub Mar 16, 2016
454363f
Fix types for the Downloader.
debbiedub Apr 16, 2016
b454282
Fixes that make all the junit tests work.
debbiedub Apr 17, 2016
2a181c1
Fixed the file reading and disk index creation.
debbiedub Apr 17, 2016
7731323
Next step in the work to restore the FreenetURI.
debbiedub Jul 2, 2017
448a714
Fixes to allow moving/linking using DownloadAll.java.
debbiedub Jul 11, 2017
1fd4803
Count the amount of URIs of different kinds.
debbiedub Jul 11, 2017
a6423db
Start from wherever.
debbiedub Jul 12, 2017
f9e0652
Progressively slower start of new fetches.
debbiedub Jul 14, 2017
95b55b6
Cleaned out old commented-out logic.
debbiedub Jul 14, 2017
9050a4a
Removed not used constructor.
debbiedub Jul 14, 2017
756d304
Removed the code that caused an empty directory first time.
debbiedub Jul 14, 2017
456ac3f
Added code to abort when a second level is added to the top term list.
debbiedub Jul 14, 2017
e417450
Removed pointless print.
debbiedub Jul 14, 2017
2d01290
Avoid race condition for ongoingUploads.
debbiedub Jul 17, 2017
c8bfad6
Put the cast back in the parsing.
debbiedub Jul 18, 2017
156d165
Simplified the logic when calculating next URI to download.
debbiedub Jul 18, 2017
1d5b178
Made Comparable and Serializable.
debbiedub Jul 23, 2017
88186a6
Abort if upload fails. Show times.
debbiedub Jul 24, 2017
d3cf20e
Use FreenetURI objects within the trees.
debbiedub Jul 24, 2017
325b454
Changed to logging of String URIs to show problems.
debbiedub Jul 24, 2017
493cc5c
Retry indefinately, if failed.
debbiedub Jul 26, 2017
6adf329
Fixed cast problem when processing subnodes.
debbiedub Jul 27, 2017
91cd50f
Reduce waiting time factor based on download jobs.
debbiedub Jul 28, 2017
0836c56
Use the library implementation of the FreenetURI also in the plugin.
debbiedub Jul 28, 2017
58f6141
Avoid creating USK for every new index entry.
debbiedub Aug 20, 2017
ba6a077
Enable tests for uploader.
debbiedub Aug 27, 2017
c3734a0
Drop terms that are parts of keys and torrent references.
debbiedub Aug 27, 2017
02cb754
Report the current edition to Spider.
debbiedub Sep 3, 2017
1bcd91d
Removed duplicate logging.
debbiedub Sep 9, 2017
ae9dc1f
Removed duplicate logging.
debbiedub Sep 9, 2017
6acc6b6
Queue pages to be uploaded.
debbiedub Oct 15, 2017
c6b13fb
Handle the case when there are no filtered or processed files.
debbiedub Nov 11, 2017
6688518
Tuned the upload speed to work with less intensity.
debbiedub Nov 15, 2017
84fdc1e
Simplified the adjustments of the download speed.
debbiedub Nov 21, 2017
b65c04a
Made the distribution of new USK versions more even.
debbiedub Nov 25, 2017
a4d2914
Refactored the DownloadAll to prepare for a one-shot.
debbiedub Dec 18, 2017
7e871b1
First version of one-shot download.
debbiedub Dec 18, 2017
8d62d9b
Work started with removal of terms.
debbiedub Dec 18, 2017
0b90c46
Reduce the days between USKs.
debbiedub Mar 3, 2018
712c228
Avoid fetching items not connected to the latest (if possible).
debbiedub Jan 17, 2018
7c91f45
Don't consider a failed upload completed.
debbiedub Mar 3, 2018
6a669e6
Renamed the old DownloadAll to a better name.
debbiedub Mar 10, 2018
e89b388
Many fixes to the DownloadAllOnce.
debbiedub Dec 30, 2017
849d199
Added tests.
debbiedub Mar 13, 2018
e3232ec
Added generics of a method.
debbiedub Mar 13, 2018
2eac8a0
Improved the statistics logging.
debbiedub Mar 17, 2018
bbafff7
Adjusted the logic for splitting files to process for upload.
debbiedub Jul 1, 2018
15ab5d8
Added logging when deferring something not connected to the last root.
debbiedub Sep 20, 2018
f2981ae
Parse a little slower.
debbiedub Sep 20, 2018
49bf793
Catch another case when shutting down the executor.
debbiedub Oct 1, 2018
9a0f62c
Remove more files every lap.
debbiedub Oct 14, 2018
a274590
Fixed some warnings.
debbiedub Dec 2, 2018
5cca0a1
For the DownloadAllOnce, remember the not available unfetchables.
debbiedub Dec 5, 2018
20fd1ca
Improved the stopping of the process.
debbiedub Dec 27, 2018
6feda2b
Improved the name of the downloading.
debbiedub Dec 30, 2018
bb7ac9a
Convert to only use tabs in indents.
debbiedub Jan 13, 2019
4034dca
Added function to specify number of simultaneous sessions.
debbiedub Jan 25, 2019
d6820dc
fix generic
Olezha Dec 10, 2019
2f4d210
explanatory comment added
Olezha Dec 10, 2019
86a7d0a
update java version to 8 and snakeyaml to 1.25
Olezha Dec 10, 2019
c5b130d
bypass for termTable
Olezha Dec 10, 2019
7ea5f86
clean YamlReaderWriter and test
Olezha Dec 10, 2019
07d43f9
refactoring: bypass for same bug in second place
Olezha Dec 10, 2019
bc7ab73
disable snakeyaml implicit resolvers
Olezha Dec 10, 2019
10d6ea2
kludge related to bc7ab73
Olezha Dec 12, 2019
afee805
clean
Olezha Dec 12, 2019
77ca196
localized the place that led to the hang + temporary solution
Olezha Dec 16, 2019
c7d15de
merge debbiedub's fcp-uploader
Olezha Jan 6, 2020
c950a78
merge uploader
Olezha Jan 8, 2020
a6a9a94
first uploader launch
Olezha Jan 8, 2020
a013ef9
continued testing
Olezha Jan 9, 2020
2ef742a
unclear null in url
Olezha Jan 14, 2020
4cd7990
try to fix hang
Olezha Jan 20, 2020
a8b8490
backward compatibility with previous indexes
Olezha Jan 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions .classpath

This file was deleted.

7 changes: 7 additions & 0 deletions .hgignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
build
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this file? The main repository is a git repository not a mercurial (AFAIK).

shared/bin
uploader/bin
lib/snakeyaml-1.5.jar
dist/Library.jar
dist/uploader.jar
shared/TermEntryTest/test.yml
17 changes: 0 additions & 17 deletions .project

This file was deleted.

7 changes: 7 additions & 0 deletions README
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Build:

plugin-Library$ ant


== Javadoc ==

If you want to generate Javadocs, download bliki-doclet, which is a little
Expand Down Expand Up @@ -56,3 +57,9 @@ this may change soon :
<input type="submit" />
</form>


== Ongoing work to split ==

The plugin is in src, test (for historical reasons).

The uploader (standalone program) is in uploader/src and uploader/test depending on fcp and plugin.
3 changes: 2 additions & 1 deletion README.clean-build
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
Library depends on SnakeYAML. It will fetch it from a known URL with a known version and checksums by default. If you want a completely clean build, fetch it yourself and build it yourself, and put it in lib/SnakeYAML-1.3.jar.
Library depends on SnakeYAML. It will fetch it from a known URL with a known version and checksums by default.
If you want a completely clean build, fetch it yourself and build it yourself, and put it in lib/SnakeYAML-1.3.jar.
26 changes: 13 additions & 13 deletions build.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
<!-- ant build file for Freenet -->

<project name="plugins-Library" default="dist" basedir=".">
<property name="freenet-cvs-snapshot.location" location="../fred/dist/freenet.jar"/>
<property name="freenet-cvs-snapshot.location" location="../fred/build/libs/freenet.jar"/>
<property name="freenet-ext.location" location="../fred/lib/freenet/freenet-ext.jar"/>
<property name="source-version" value="1.6"/>
<property name="target-version" value="1.6"/>
<property name="source-version" value="1.8"/>
<property name="target-version" value="1.8"/>
<property name="build" location="build/"/>
<property name="build-test" location="build-test/"/>
<property name="run-test" location="run-test/"/>
Expand Down Expand Up @@ -64,14 +64,14 @@
<delete dir="${tmp}"/>
</target>

<property name="SnakeYAML.mirror" value="https://downloads.freenetproject.org/snakeyaml-1.5.jar" />
<property name="SnakeYAML.md5" value="1788db5d529a981b090632b8e49b8cd2" />
<property name="SnakeYAML.sha" value="a3654035d0b68db775034afa5425a62250de8184" />
<property name="SnakeYAML.mirror" value="https://repo1.maven.org/maven2/org/yaml/snakeyaml/1.25/snakeyaml-1.25.jar" />
<property name="SnakeYAML.md5" value="6f7d5b8f596047aae07a3bf6f23a0bf2" />
<property name="SnakeYAML.sha" value="8b6e01ef661d8378ae6dd7b511a7f2a33fae1421" />
<available property="SnakeYAML.exist" classname="org.yaml.snakeyaml.Yaml" classpathref="lib.path"/>
<target name="SnakeYAML-get" unless="SnakeYAML.exist">
<get verbose="true" src="${SnakeYAML.mirror}" dest="${tmp}/snakeyaml-1.5.jar" />
<checksum file="${tmp}/snakeyaml-1.5.jar" algorithm="MD5" property="${SnakeYAML.md5}" verifyProperty="SnakeYAMLMD5ok" />
<checksum file="${tmp}/snakeyaml-1.5.jar" algorithm="SHA" property="${SnakeYAML.sha}" verifyProperty="SnakeYAMLSHAok" />
<get verbose="true" src="${SnakeYAML.mirror}" dest="${tmp}/snakeyaml-1.25.jar" />
<checksum file="${tmp}/snakeyaml-1.25.jar" algorithm="MD5" property="${SnakeYAML.md5}" verifyProperty="SnakeYAMLMD5ok" />
<checksum file="${tmp}/snakeyaml-1.25.jar" algorithm="SHA" property="${SnakeYAML.sha}" verifyProperty="SnakeYAMLSHAok" />
<fail message="SnakeYAML.zip checksum mismatch">
<condition>
<or>
Expand All @@ -80,7 +80,7 @@
</or>
</condition>
</fail>
<move file="${tmp}/snakeyaml-1.5.jar" todir="lib/" />
<move file="${tmp}/snakeyaml-1.25.jar" todir="lib/" />
</target>

<!--<property name="KeyExplorer.mirror" value="http://cloud.github.com/downloads/freenet/plugin-Library-staging/KeyExplorer-dacfafecbc82aecdeffa56bef4a047a7f6c7f08d.jar" />
Expand Down Expand Up @@ -131,7 +131,7 @@
</javac>

<!-- Force compile of Version.java in case compile of ${src} didn't trigger it -->
<javac srcdir="${build}" destdir="${build}" debug="on" optimize="on" source="1.5" target="1.5">
<javac srcdir="${build}" destdir="${build}" debug="on" optimize="on" source="1.8" target="1.8">
<classpath>
<path refid="lib.path"/>
</classpath>
Expand All @@ -146,7 +146,7 @@
</target>

<target name="compile-tester" depends="compile" if="tester">
<javac srcdir="test/" destdir="${build}" debug="on" optimize="on" source="1.5" target="1.5">
<javac srcdir="test/" destdir="${build}" debug="on" optimize="on" source="1.8" target="1.8">
<classpath>
<path refid="lib.path"/>
<!-- Uses some code that uses junit -->
Expand All @@ -173,7 +173,7 @@

<!-- ================================================== -->
<target name="unit-build" depends="compile" if="junit.present" unless="skip_tests">
<javac srcdir="test/" destdir="${build-test}" debug="on" optimize="on" source="1.5">
<javac srcdir="test/" destdir="${build-test}" debug="on" optimize="on" source="1.8">
<classpath>
<path refid="lib.path"/>
<pathelement path="${build}"/>
Expand Down
7 changes: 7 additions & 0 deletions loop.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/sh -x
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file for local tests? Can we clarify it if so?


while test -f library.continue.loop
do
java -jar `dirname $0`/dist/uploader.jar
sleep 60
done
18 changes: 18 additions & 0 deletions src/plugins/Library/ArchiverFactory.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package plugins.Library;

import plugins.Library.io.ObjectStreamReader;
import plugins.Library.io.ObjectStreamWriter;
import plugins.Library.io.serial.LiveArchiver;
import plugins.Library.util.exec.SimpleProgress;

public interface ArchiverFactory {
<T, S extends ObjectStreamWriter & ObjectStreamReader>
LiveArchiver<T, SimpleProgress>
newArchiver(S rw, String mime, int size,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Methods in interfaces should have at least rudimentary documentation, otherwise nobody will ever know what that int is meant to be…

Priority priorityLevel);

<T, S extends ObjectStreamWriter & ObjectStreamReader>
LiveArchiver<T, SimpleProgress>
newArchiver(S rw, String mime, int size,
LiveArchiver<T, SimpleProgress> archiver);
}
14 changes: 14 additions & 0 deletions src/plugins/Library/FactoryRegister.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package plugins.Library;

public class FactoryRegister {
private static ArchiverFactory archiver = null;

public static void register(ArchiverFactory factory) {
archiver = factory;
}

public static ArchiverFactory getArchiverFactory() {
assert archiver != null;
return archiver;
}
}
Loading