Redwerk Work#19

Open

ArneBab wants to merge 189 commits intohyphanet:masterfrom

redwerk:frn-97-4-build-uploader

Commits on Dec 19, 2015

Removed dead code.

Debora Wöpcke
authored and
debbiedub
committed
More code cleanups.

Debora Wöpcke
authored and
debbiedub
committed
Setting up the structure.

anonymous
authored and
debbiedub
committed
Ignore built files.

anonymous
authored and
debbiedub
committed
Second part of the moving.

anonymous
authored and
debbiedub
committed
Moving another directory.

anonymous
authored and
debbiedub
committed
Moved most of plugins.Library.util.concurrent.

anonymous
authored and
debbiedub
committed
Moved the rest of plugin.Library.util.concurrent.

anonymous
authored and
debbiedub
committed
Moved first (and simple) part of the plugins.Library.util.

anonymous
authored and
debbiedub
committed
Moved two more classes from plugins.Library.util.

anonymous
authored and
debbiedub
committed
Moved some serialisers from plugins.Library.io.serial.

anonymous
authored and
debbiedub
committed
Cleanups to allow moving of the classes.

anonymous
authored and
debbiedub
committed
Moved the rest of the classes from plugins.Library.util.

anonymous
authored and
debbiedub
committed
Removed logging to allow for the move.

anonymous
authored and
debbiedub
committed

Commits on Feb 20, 2016

Moved almost all files from plugins.Library.io and .serial.

anonymous
authored and
debbiedub
committed
Refactoring the yaml-code to factor out the Freenet classes.

anonymous
authored and
debbiedub
committed
Moved the reader/writer and entries.

anonymous
authored and
debbiedub
committed
Re-enabled compilation using ant.

anonymous
authored and
debbiedub
committed
Fixes of first problem found when running on old index.

anonymous
authored and
debbiedub
committed
Refactored to simplify code and include object reading time in the logged time.

anonymous
authored and
debbiedub
committed
Added more stack traces in the log to be able to spot complex problems.

anonymous
authored and
debbiedub
committed
Reverted the type changes in the TermPageEntry to allow old indexes to be read.

anonymous
authored and
debbiedub
committed
Some eclipse settings for uploader.

anonymous
authored and
debbiedub
committed
Some small simple refactorings to prepare moving ProtoIndex to shared part.

anonymous
authored and
debbiedub
committed
Changed FreenetURI to String in index/Proto*.java.

anonymous
authored and
debbiedub
committed
Moved Index.java interface and URIEntry, URIKey to the shared.

anonymous
authored and
debbiedub
committed
Created a register for the factory for LiveArchiver.

anonymous
authored and
debbiedub
committed
Moved the ProtoIndex to shared.

anonymous
authored and
debbiedub
committed
First framework of the uploader compiling.

anonymous
authored and
debbiedub
committed
Create the uploader jar.

anonymous
authored and
debbiedub
committed
Reordered the removelistener to reduce concurrentexceptions.

anonymous
authored and
debbiedub
committed
Added spider compatibility when reading the terms.

anonymous
authored and
debbiedub
committed
Improved the proof of concept.

anonymous
authored and
debbiedub
committed
Build more packages.

anonymous
authored and
debbiedub
committed
Some more work, slowly approaching uploading.

anonymous
authored and
debbiedub
committed
New workspace with project named a little differently.

anonymous
authored and
debbiedub
committed
Added debug trace.

anonymous
authored and
debbiedub
committed
Abandoned the idea of separating the Archiver and joined together to

anonymous
authored and
debbiedub
committed
Added upload to the fcp-adapter.

anonymous
authored and
debbiedub
committed
Removed tabs.

anonymous
authored and
debbiedub
committed
Use the file the second time, to avoid locking up the writer.

anonymous
authored and
debbiedub
committed
Second attempt at removing the tabs.

anonymous
authored and
debbiedub
committed
Moved the handling of the close-detection to get the right close message.

anonymous
authored and
debbiedub
committed
Created the processing of files and creating the on file index.

anonymous
authored and
debbiedub
committed
Let also this pool go down to 0 threads when idle so that the script can eventually finish.

anonymous
authored and
debbiedub
committed
Tune the parameters for amount of stored entries and amount of files.

anonymous
authored and
debbiedub
committed
Don't finish the operation until the PutSuccessful is received.

anonymous
authored and
debbiedub
committed
First working implementation of the IndexPeeker to group Terms.

anonymous
authored and
debbiedub
committed
Refactored the interface to the IndexPeeker.

anonymous
authored and
debbiedub
committed
Removed not used code and imports.

anonymous
authored and
debbiedub
committed
Removed not used field.

anonymous
authored and
debbiedub
committed
Implementation that does the actual peeking in the Index to select only entries at a single location in the tree.

anonymous
authored and
debbiedub
committed
Made the directory creator resilient against already existing directories.

anonymous
authored and
debbiedub
committed
Removed confusing trace message.

anonymous
authored and
debbiedub
committed
Let the files be a little bigger. (1000000 entries ~ 200M).

anonymous
authored and
debbiedub
committed
Buggfix: delay deflating the tree until it is all filled.

anonymous
authored and
debbiedub
committed
Refactored IndexPeeker to make it easy to do more section at the same time.

anonymous
authored and
debbiedub
committed
Factored out the setting up and taking down of the Fcp session.

anonymous
authored and
debbiedub
committed
Buggfix for the non-session case.

anonymous
authored and
debbiedub
committed
Changed the handling of files to process all not previously terms each time a selected is processed.

anonymous
authored and
debbiedub
committed
Made it possible to specify the client name to FcpSession.

anonymous
authored and
debbiedub
committed
Process the files in the correct order.

anonymous
authored and
debbiedub
committed
Factored out the FileClientPutter.

anonymous
authored and
debbiedub
committed
Moved the cleanup thread entirely into the FileClientPutter.

anonymous
authored and
debbiedub
committed
Reverted back to 1148 since it didn't work.

anonymous
authored and
debbiedub
committed
Initial revision of an index downloader.

anonymous
authored and
debbiedub
committed
Improved code quality.

anonymous
authored and
debbiedub
committed
New calculation of estimate of entire tree.

anonymous
authored and
debbiedub
committed
Added function to include the contents of several selected file, if small.

anonymous
authored and
debbiedub
committed
Adjustments in the calculation of how many selected to include.

anonymous
authored and
debbiedub
committed
Calculate how many uploads that are resurrected.

anonymous
authored and
debbiedub
committed
The new known good getter changed name in FCPlib.

anonymous
authored and
debbiedub
committed
Attempted to tune to a better approximation of the size of an upload.

anonymous
authored and
debbiedub
committed
Changed the eclipse project names and location of the fcp lib.

anonymous
authored and
debbiedub
committed
Updated the Eclipse project name.
debbiedub
committed
Slow down the build up of the queue of uploads.
debbiedub
committed
Tuned the size of the uploads down a bit.
debbiedub
committed
Reduce ant warnings.
debbiedub
committed
Moved the index selection to a separate web page.
debbiedub
committed
Force my testing index into the mix.
debbiedub
committed
Add retrieval of index of sites for a word.
debbiedub
committed
Reduced logging when downloading.
debbiedub
committed
Script to run the upload perpetually.
debbiedub
committed
Corrected the logic for adding multiple selected in an upload.
debbiedub
committed
Take down the uploader jobs service.
debbiedub
committed
Add log messages when URIs are not fetched.
debbiedub
committed
Moved the pruning of the trees to avoid always having at least two of them.
debbiedub
committed
End the waiting indefinately.
debbiedub
committed
Saving the unpacked object for examining it in the debugger.
debbiedub
committed
Fixed problem with specific downloads. Added more logging.
debbiedub
committed

Commits on Apr 16, 2016

Accept and count that the CHK:s has occasionally changed.
debbiedub
committed

Commits on Apr 27, 2016

Adjusted the logging.
debbiedub
committed

Commits on May 1, 2016

Reduce the amount of downloading while there is uploading.
debbiedub
committed

Commits on May 4, 2016

Added max queue-len.
debbiedub
committed

Commits on May 14, 2016

Commits on May 15, 2016

Commits on Jun 10, 2016

Fixed some exceptions. Adjusted logging.
debbiedub
committed

Commits on Aug 16, 2016

Factored out aging of running fetches.
debbiedub
committed

Commits on Aug 24, 2016

Commits on Nov 5, 2016

Removed callback message.
debbiedub
committed

Commits on Jul 12, 2017

Start from wherever.
debbiedub
committed

Commits on Jul 17, 2017

Avoid race condition for ongoingUploads.
debbiedub
committed

Commits on Jul 18, 2017

Commits on Jul 23, 2017

Made Comparable and Serializable.
debbiedub
committed

Commits on Jul 24, 2017

Commits on Jul 26, 2017

Retry indefinately, if failed.
debbiedub
committed

Commits on Jul 27, 2017

Fixed cast problem when processing subnodes.
debbiedub
committed

Commits on Jul 28, 2017

Commits on Aug 20, 2017

Avoid creating USK for every new index entry.
debbiedub
committed

Commits on Aug 27, 2017

Commits on Sep 3, 2017

Report the current edition to Spider.
debbiedub
committed

Commits on Sep 9, 2017

Commits on Oct 15, 2017

Queue pages to be uploaded.
debbiedub
committed

Commits on Nov 11, 2017

Handle the case when there are no filtered or processed files.
debbiedub
committed

Commits on Nov 15, 2017

Tuned the upload speed to work with less intensity.
debbiedub
committed

Commits on Nov 21, 2017

Simplified the adjustments of the download speed.
debbiedub
committed

Commits on Dec 9, 2017

Made the distribution of new USK versions more even.
debbiedub
committed

Commits on Dec 27, 2017

Commits on Mar 17, 2018

Improved the statistics logging.
debbiedub
committed

Commits on Jul 1, 2018

Adjusted the logic for splitting files to process for upload.
debbiedub
committed

Commits on Sep 20, 2018

Commits on Oct 1, 2018

Catch another case when shutting down the executor.
debbiedub
committed

Commits on Oct 14, 2018

Remove more files every lap.
debbiedub
committed

Commits on Dec 2, 2018

Fixed some warnings.
debbiedub
committed

Commits on Dec 8, 2018

For the DownloadAllOnce, remember the not available unfetchables.
debbiedub
committed

Commits on Dec 27, 2018

Improved the stopping of the process.
debbiedub
committed

Commits on Dec 30, 2018

Improved the name of the downloading.
debbiedub
committed

Commits on Jan 13, 2019

Convert to only use tabs in indents.
debbiedub
committed

Commits on Feb 24, 2019

Added function to specify number of simultaneous sessions.
debbiedub
committed

Commits on Dec 12, 2019

Commits on Dec 16, 2019

localized the place that led to the hang + temporary solution
Olezha
committed

Commits on Jan 6, 2020

merge debbiedub's fcp-uploader
Olezha
committed

Commits on Jan 8, 2020

Commits on Jan 9, 2020

continued testing
Olezha
committed

Commits on Jan 14, 2020

unclear null in url
Olezha
committed

Commits on Jan 15, 2020

build uploader
Olezha
committed

Commits on Jan 20, 2020

try to fix hang
Olezha
committed

Commits on Jan 30, 2020

Commits on Mar 17, 2020

Update build
Olezha
committed

Commits on Mar 18, 2020

Clean
Olezha
committed
Clarify process
Olezha
committed

Commits on Mar 23, 2020