From 580b2f67f2a59384773464eb7550b3b8936afd20 Mon Sep 17 00:00:00 2001 From: Caleb Maclennan Date: Wed, 11 Jul 2012 12:08:42 +0300 Subject: [PATCH 001/166] Minor English cleanup. --- README | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/README b/README index a15f1c9b..65597ce4 100644 --- a/README +++ b/README @@ -3,20 +3,20 @@ Grive 0.2.0 http://www.lbreda.com/grive/ -Grive is still considered experimental. It just downloads all the files in your google drive -into the current directory. After you make some changes to the local files, run grive and -it will upload your changes back to your google drive. New files created in local or google -drive will be uploaded or downloaded respectively. Deleted files will also be "removed". +Grive is still considered experimental. It simply downloads all the files in your Google Drive +into the current directory. After you make some changes to the local files, run grive again and +it will upload your changes back to your Google Drive. New files created locally or in Google +Drive will be uploaded or downloaded respectively. Deleted files will also be "removed". Currently Grive will NOT destroy any of your files: it will only move the files to a -directory named .trash, or put it in the google drive trash. You can always recover them. +directory named .trash or put them in the Google Drive trash. You can always recover them. -There are a few things that grive does not do at the moment: -- wait for changes in file system to occur and upload. Grive only sync when you run it. +There are a few things that Grive does not do at the moment: +- wait for changes in file system to occur and upload. A sync is only performed when you run Grive. - symbolic links support - support for Google documents - support for files >2GB -Of course these will be done in future, possibly the next release. +Of course these will be added in the future, possibly the next release. You need the following libraries: @@ -35,16 +35,16 @@ Grive uses cmake to build, see the instructions in: http://www.lbreda.com/grive/installation -for detailed procedures to compile Grive. +...for detailed procedures to compile Grive. -When grive is ran for the first time, you should use the "-a" argument to grant -permission to grive to access to your Google Drive. An URL should be printed. -Go to the link. You will need to login to your google account if you haven't -done so. After granting the permission to grive, the browser will show you -an authenication code. Copy-and-paste that to the standard input of grive. +When Grive is run for the first time, you should use the "-a" argument to grant +permission to Grive to access to your Google Drive. A URL should be printed. +Go to the link. You will need to login to your Google account if you haven't +done so. After granting the permission to Grive, the browser will show you +an authenication code. Copy-and-paste that to the standard input of Grive. -If everything works fine, grive will create a .grive and a .grive_state file in your +If everything works fine, Grive will create .grive and .grive_state files in your current directory. It will also start downloading files from your Google Drive to your current directory. -Enjoy! +Enjoy! \ No newline at end of file From 66b3d37872dc8f8f4a28691f02e152e0aa32a8b9 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Wed, 11 Jul 2012 22:51:08 +0800 Subject: [PATCH 002/166] README: beta quality --- README | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README b/README index 65597ce4..5130b110 100644 --- a/README +++ b/README @@ -3,10 +3,10 @@ Grive 0.2.0 http://www.lbreda.com/grive/ -Grive is still considered experimental. It simply downloads all the files in your Google Drive -into the current directory. After you make some changes to the local files, run grive again and -it will upload your changes back to your Google Drive. New files created locally or in Google -Drive will be uploaded or downloaded respectively. Deleted files will also be "removed". +Grive can be considered still considered beta quality. It simply downloads all the files in your +Google Drive into the current directory. After you make some changes to the local files, run +grive again and it will upload your changes back to your Google Drive. New files created locally +or in Google Drive will be uploaded or downloaded respectively. Deleted files will also be "removed". Currently Grive will NOT destroy any of your files: it will only move the files to a directory named .trash or put them in the Google Drive trash. You can always recover them. @@ -47,4 +47,4 @@ If everything works fine, Grive will create .grive and .grive_state files in you current directory. It will also start downloading files from your Google Drive to your current directory. -Enjoy! \ No newline at end of file +Enjoy! From 278037b07b35c57fd5e86ea74f6f157e333873c8 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 12 Jul 2012 00:30:19 +0800 Subject: [PATCH 003/166] fixed incorrect name in pdf files --- libgrive/src/drive/Entry.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libgrive/src/drive/Entry.cc b/libgrive/src/drive/Entry.cc index a04ca28b..d60d3372 100644 --- a/libgrive/src/drive/Entry.cc +++ b/libgrive/src/drive/Entry.cc @@ -192,7 +192,7 @@ bool Entry::IsRemoved() const std::string Entry::Name() const { - return m_kind == "file" ? m_filename : m_title ; + return (m_kind == "file" || m_kind == "pdf") ? m_filename : m_title ; } } // end of namespace From 80f3fbf37055baa038339d6f633178fc6fdba1d2 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 12 Jul 2012 00:36:58 +0800 Subject: [PATCH 004/166] refactored the logs to debug sync problems --- libgrive/src/drive/Drive.cc | 28 ++++++++++++--------- libgrive/src/drive/Feed.cc | 43 +++++++++++++++++++++++++++----- libgrive/src/drive/Feed.hh | 16 +++++++++++- libgrive/src/drive/Resource.cc | 3 +++ libgrive/src/drive/State.cc | 1 - libgrive/src/http/ResponseLog.cc | 37 ++++++++++++++++++++++++--- libgrive/src/http/ResponseLog.hh | 7 +++++- 7 files changed, 111 insertions(+), 24 deletions(-) diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index 258ce94d..f0d454f1 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -139,28 +139,32 @@ void Drive::DetectChanges() SyncFolders( &http ) ; Log( "Reading remote server file list", log::info ) ; - http::XmlResponse xrsp ; - http.Get( feed_base + "?showfolders=true&showroot=true", &xrsp, m_http_hdr ) ; - xml::Node resp = xrsp.Response() ; - - m_resume_link = resp["link"]. + Feed feed ; +// feed.EnableLog( "/tmp/file", ".xml" ) ; + feed.Start( &http, m_http_hdr, feed_base + "?showfolders=true&showroot=true" ) ; + + m_resume_link = feed.Root()["link"]. Find( "@rel", "http://schemas.google.com/g/2005#resumable-create-media" )["@href"] ; - - Feed feed( resp ) ; + do { - std::for_each( feed.begin(), feed.end(), boost::bind( &Drive::FromRemote, this, _1 ) ) ; + std::for_each( + feed.begin(), feed.end(), + boost::bind( &Drive::FromRemote, this, _1 ) ) ; + } while ( feed.GetNext( &http, m_http_hdr ) ) ; // pull the changes feed if ( prev_stamp != -1 ) { - http::ResponseLog log( "/tmp/changes-", ".xml", &xrsp ) ; Log( "Detecting changes from last sync", log::info ) ; - http.Get( ChangesFeed(prev_stamp+1), &log, m_http_hdr ) ; + Feed changes ; +// feed.EnableLog( "/tmp/changes", ".xml" ) ; + feed.Start( &http, m_http_hdr, ChangesFeed(prev_stamp+1) ) ; - Feed changes( xrsp.Response() ) ; - std::for_each( changes.begin(), changes.end(), boost::bind( &Drive::FromChange, this, _1 ) ) ; + std::for_each( + changes.begin(), changes.end(), + boost::bind( &Drive::FromChange, this, _1 ) ) ; } } diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed.cc index bbb1b7f6..72bfa9ac 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed.cc @@ -20,13 +20,20 @@ #include "Feed.hh" #include "http/Agent.hh" +#include "http/ResponseLog.hh" #include "http/XmlResponse.hh" #include "xml/NodeSet.hh" +#include + #include namespace gr { +Feed::Feed( ) +{ +} + Feed::Feed( const xml::Node& root ) : m_root ( root ), m_entries ( m_root["entry"] ) @@ -55,6 +62,22 @@ std::string Feed::Next() const return nss.empty() ? "" : std::string(nss["@href"]) ; } +void Feed::Start( http::Agent *http, const http::Header& auth, const std::string& url ) +{ + http::XmlResponse xrsp ; + http::ResponseLog log( &xrsp ) ; + + if ( m_log.get() != 0 ) + log.Reset( + (boost::format( "%1%-%2%." ) % m_log->prefix % m_log->sequence++).str(), + m_log->suffix, &xrsp ) ; + + http->Get( url, &log, auth ) ; + + m_root = xrsp.Response() ; + m_entries = m_root["entry"] ; +} + bool Feed::GetNext( http::Agent *http, const http::Header& auth ) { assert( http != 0 ) ; @@ -62,12 +85,7 @@ bool Feed::GetNext( http::Agent *http, const http::Header& auth ) xml::NodeSet nss = m_root["link"].Find( "@rel", "next" ) ; if ( !nss.empty() ) { - http::XmlResponse xrsp ; - http->Get( nss["@href"], &xrsp, auth ) ; - - m_root = xrsp.Response() ; - m_entries = m_root["entry"] ; - + Start( http, auth, nss["@href"] ) ; return true ; } else @@ -90,4 +108,17 @@ Feed::iterator::reference Feed::iterator::dereference() const return Entry( *base_reference() ) ; } +void Feed::EnableLog( const std::string& prefix, const std::string& suffix ) +{ + m_log.reset( new LogInfo ) ; + m_log->prefix = prefix ; + m_log->suffix = suffix ; + m_log->sequence = 0 ; +} + +const xml::Node& Feed::Root() const +{ + return m_root ; +} + } // end of namespace diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/drive/Feed.hh index 321498cc..f41fb7d3 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/drive/Feed.hh @@ -43,15 +43,29 @@ public : public : explicit Feed( const xml::Node& root ) ; + Feed( ) ; + void Start( http::Agent *http, const http::Header& auth, const std::string& url ) ; + void Assign( const xml::Node& root ) ; + const xml::Node& Root() const ; iterator begin() const ; iterator end() const ; std::string Next() const ; bool GetNext( http::Agent *http, const http::Header& auth ) ; - + + void EnableLog( const std::string& prefix, const std::string& suffix ) ; + private : + struct LogInfo + { + std::string prefix ; + std::string suffix ; + std::size_t sequence ; + } ; + std::auto_ptr m_log ; + xml::Node m_root ; xml::NodeSet m_entries ; } ; diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index a4e85013..4c7e8136 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -259,7 +259,10 @@ void Resource::FromLocal( const DateTime& last_sync ) // remote_deleted first, it will be updated to sync/remote_changed // in FromRemote() else + { +// Trace( "file %1% mtime %2%, last_sync %3%", path, m_mtime, last_sync ) ; m_state = ( m_mtime > last_sync ? local_new : remote_deleted ) ; + } m_name = Path2Str( path.filename() ) ; m_kind = fs::is_directory(path) ? "folder" : "file" ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index dddb0128..373e1b38 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -73,7 +73,6 @@ void State::FromLocal( const fs::path& p, gr::Resource* folder ) for ( fs::directory_iterator i( p ) ; i != fs::directory_iterator() ; ++i ) { std::string fname = Path2Str(i->path().filename()) ; -// Trace( "found file %1%", i->path() ) ; if ( IsIgnore(fname) ) Log( "file %1% is ignored by grive", fname, log::verbose ) ; diff --git a/libgrive/src/http/ResponseLog.cc b/libgrive/src/http/ResponseLog.cc index 0e7e631e..8de81fe4 100644 --- a/libgrive/src/http/ResponseLog.cc +++ b/libgrive/src/http/ResponseLog.cc @@ -29,14 +29,28 @@ ResponseLog::ResponseLog( const std::string& prefix, const std::string& suffix, Receivable *next ) : - m_log( Filename(prefix, suffix).c_str() ), - m_next( next ) + m_enabled ( true ), + m_log ( Filename(prefix, suffix).c_str() ), + m_next ( next ) { + assert( m_next != 0 ) ; +} + +ResponseLog::ResponseLog( Receivable *next ) : + m_enabled ( false ), + m_next ( next ) +{ + assert( m_next != 0 ) ; } std::size_t ResponseLog::OnData( void *data, std::size_t count ) { - m_log.rdbuf()->sputn( reinterpret_cast(data), count ) ; + if ( m_enabled ) + { + assert( m_log.rdbuf() != 0 ) ; + m_log.rdbuf()->sputn( reinterpret_cast(data), count ) ; + } + return m_next->OnData( data, count ) ; } @@ -51,4 +65,21 @@ std::string ResponseLog::Filename( const std::string& prefix, const std::string& return prefix + DateTime::Now().Format( "%H%M%S" ) + suffix ; } +void ResponseLog::Enable( bool enable ) +{ + m_enabled = enable ; +} + +void ResponseLog::Reset( const std::string& prefix, const std::string& suffix, Receivable *next ) +{ + assert( next != 0 ) ; + + if ( m_log.is_open() ) + m_log.close() ; + + m_log.open( Filename(prefix, suffix).c_str() ) ; + m_next = next ; + m_enabled = true ; +} + }} // end of namespace diff --git a/libgrive/src/http/ResponseLog.hh b/libgrive/src/http/ResponseLog.hh index a4273244..9b7de656 100644 --- a/libgrive/src/http/ResponseLog.hh +++ b/libgrive/src/http/ResponseLog.hh @@ -33,14 +33,19 @@ public : const std::string& prefix, const std::string& suffix, Receivable *next ) ; - + ResponseLog( Receivable *next ) ; + std::size_t OnData( void *data, std::size_t count ) ; void Clear() ; + void Enable( bool enable = true ) ; + void Reset( const std::string& prefix, const std::string& suffix, Receivable *next ) ; + private : static std::string Filename( const std::string& prefix, const std::string& suffix ) ; private : + bool m_enabled ; std::ofstream m_log ; Receivable *m_next ; } ; From e28fe96260636441e043120394602f5ba79779ad Mon Sep 17 00:00:00 2001 From: justin Date: Sun, 15 Jul 2012 17:33:33 +0100 Subject: [PATCH 005/166] fix deleted local file downloaded again --- libgrive/src/drive/Resource.cc | 23 ++++++++++++----------- libgrive/src/drive/Resource.hh | 10 +++++----- libgrive/src/drive/State.cc | 16 ++++++++++++++-- 3 files changed, 31 insertions(+), 18 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 4c7e8136..e253cac6 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -360,20 +360,20 @@ Resource* Resource::FindChild( const std::string& name ) } // try to change the state to "sync" -void Resource::Sync( http::Agent *http, const http::Header& auth ) +void Resource::Sync( http::Agent *http, const http::Header& auth, DateTime& sync_time ) { assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced - SyncSelf( http, auth ) ; + SyncSelf( http, auth, sync_time ) ; // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) std::for_each( m_child.begin(), m_child.end(), - boost::bind( &Resource::Sync, _1, http, auth ) ) ; + boost::bind( &Resource::Sync, _1, http, auth, boost::ref(sync_time) ) ) ; } -void Resource::SyncSelf( http::Agent* http, const http::Header& auth ) +void Resource::SyncSelf( http::Agent* http, const http::Header& auth, DateTime& sync_time ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync assert( IsRoot() || http == 0 || fs::is_directory( m_parent->Path() ) ) ; @@ -387,7 +387,7 @@ void Resource::SyncSelf( http::Agent* http, const http::Header& auth ) case local_new : Log( "sync %1% doesn't exist in server, uploading", path, log::info ) ; - if ( http != 0 && Create( http, auth ) ) + if ( http != 0 && Create( http, auth, sync_time ) ) m_state = sync ; break ; @@ -399,7 +399,7 @@ void Resource::SyncSelf( http::Agent* http, const http::Header& auth ) case local_changed : Log( "sync %1% changed in local. uploading", path, log::info ) ; - if ( http != 0 && EditContent( http, auth ) ) + if ( http != 0 && EditContent( http, auth, sync_time ) ) m_state = sync ; break ; @@ -512,7 +512,7 @@ void Resource::Download( http::Agent* http, const fs::path& file, const http::He } } -bool Resource::EditContent( http::Agent* http, const http::Header& auth ) +bool Resource::EditContent( http::Agent* http, const http::Header& auth, DateTime& sync_time ) { assert( http != 0 ) ; assert( m_parent != 0 ) ; @@ -525,10 +525,10 @@ bool Resource::EditContent( http::Agent* http, const http::Header& auth ) return false ; } - return Upload( http, m_edit, auth, false ) ; + return Upload( http, m_edit, auth, false, sync_time ) ; } -bool Resource::Create( http::Agent* http, const http::Header& auth ) +bool Resource::Create( http::Agent* http, const http::Header& auth, DateTime& sync_time ) { assert( http != 0 ) ; assert( m_parent != 0 ) ; @@ -558,7 +558,7 @@ bool Resource::Create( http::Agent* http, const http::Header& auth ) } else if ( !m_parent->m_create.empty() ) { - return Upload( http, m_parent->m_create + "?convert=false", auth, true ) ; + return Upload( http, m_parent->m_create + "?convert=false", auth, true, sync_time ) ; } else { @@ -567,7 +567,7 @@ bool Resource::Create( http::Agent* http, const http::Header& auth ) } } -bool Resource::Upload( http::Agent* http, const std::string& link, const http::Header& auth, bool post ) +bool Resource::Upload( http::Agent* http, const std::string& link, const http::Header& auth, bool post, DateTime& sync_time ) { assert( http != 0 ) ; @@ -611,6 +611,7 @@ bool Resource::Upload( http::Agent* http, const std::string& link, const http::H http->Put( uplink, data, &xml, uphdr ) ; AssignIDs( Entry( xml.Response() ) ) ; + sync_time = std::max(Entry(xml.Response()).MTime(), sync_time); return true ; } diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index b14e7ee7..8a76017c 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -77,7 +77,7 @@ public : void FromRemote( const Entry& remote, const DateTime& last_sync ) ; void FromLocal( const DateTime& last_sync ) ; - void Sync( http::Agent* http, const http::Header& auth ) ; + void Sync( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; // children access iterator begin() const ; @@ -125,9 +125,9 @@ private : void SetState( State new_state ) ; void Download( http::Agent* http, const fs::path& file, const http::Header& auth ) const ; - bool EditContent( http::Agent* http, const http::Header& auth ) ; - bool Create( http::Agent* http, const http::Header& auth ) ; - bool Upload( http::Agent* http, const std::string& link, const http::Header& auth, bool post ) ; + bool EditContent( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; + bool Create( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; + bool Upload( http::Agent* http, const std::string& link, const http::Header& auth, bool post, DateTime& sync_time ) ; void FromRemoteFolder( const Entry& remote, const DateTime& last_sync ) ; void FromRemoteFile( const Entry& remote, const DateTime& last_sync ) ; @@ -136,7 +136,7 @@ private : void DeleteRemote( http::Agent* http, const http::Header& auth ) ; void AssignIDs( const Entry& remote ) ; - void SyncSelf( http::Agent* http, const http::Header& auth ) ; + void SyncSelf( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; private : std::string m_name ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 373e1b38..280c071f 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -264,8 +264,20 @@ void State::Write( const fs::path& filename ) const void State::Sync( http::Agent *http, const http::Header& auth ) { - m_res.Root()->Sync( http, auth ) ; - m_last_sync = DateTime::Now() ; + // set the last sync time from the time returned by the server for the last file synced + // if the sync time hasn't changed (i.e. now files have been uploaded) + // set the last sync time to the time on the client + // ideally because we compare server file times to the last sync time + // the last sync time would always be a server time rather than a client time + // TODO - WARNING - do we use the last sync time to compare to client file times + // need to check if this introduces a new problem + DateTime last_sync_time = m_last_sync; + m_res.Root()->Sync( http, auth, last_sync_time ) ; + if (last_sync_time == m_last_sync) { + m_last_sync = DateTime::Now(); + } else { + m_last_sync = last_sync_time; + } } long State::ChangeStamp() const From 394982da4efd5da938daf7ed6769ce2c7f8623f9 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 16 Jul 2012 22:30:30 +0800 Subject: [PATCH 006/166] added _FILE_OFFSET_BITS=64 (#88) --- CMakeLists.txt | 5 +++++ grive/CMakeLists.txt | 2 -- libgrive/CMakeLists.txt | 1 - 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 92c50b4b..ca39cb19 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,6 +1,11 @@ cmake_minimum_required(VERSION 2.8) +# Grive version. remember to update it for every new release! set( GRIVE_VERSION "0.2.0" ) +# common compile options +add_definitions( -DVERSION="${GRIVE_VERSION}" ) +add_definitions( -D_FILE_OFFSET_BITS=64 ) + add_subdirectory( libgrive ) add_subdirectory( grive ) diff --git a/grive/CMakeLists.txt b/grive/CMakeLists.txt index cde35784..d5f8ebef 100644 --- a/grive/CMakeLists.txt +++ b/grive/CMakeLists.txt @@ -7,8 +7,6 @@ include_directories( ${OPT_INCS} ) -add_definitions( -DVERSION="${GRIVE_VERSION}" ) - file (GLOB GRIVE_EXE_SRC ${grive_SOURCE_DIR}/src/*.cc ) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 000d898b..a19fc992 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -68,7 +68,6 @@ file (GLOB LIBGRIVE_SRC ) add_definitions( - -DVERSION="${GRIVE_VERSION}" -DTEST_DATA="${libgrive_SOURCE_DIR}/test/data/" -DSRC_DIR="${libgrive_SOURCE_DIR}/src" ) From 593c335a5e408e3644efac5e2e91a82df5d3b7b9 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 16 Jul 2012 23:23:08 +0800 Subject: [PATCH 007/166] improved logging for HTTP responses --- libgrive/src/drive/Drive.cc | 4 ++-- libgrive/src/drive/Feed.cc | 5 +++-- libgrive/src/drive/Resource.cc | 9 +++++++-- libgrive/src/http/ResponseLog.cc | 24 +++++++++++++++++++----- libgrive/src/util/log/Log.hh | 6 ++++++ 5 files changed, 37 insertions(+), 11 deletions(-) diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index f0d454f1..f83bdd64 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -140,7 +140,7 @@ void Drive::DetectChanges() Log( "Reading remote server file list", log::info ) ; Feed feed ; -// feed.EnableLog( "/tmp/file", ".xml" ) ; + feed.EnableLog( "/tmp/file", ".xml" ) ; feed.Start( &http, m_http_hdr, feed_base + "?showfolders=true&showroot=true" ) ; m_resume_link = feed.Root()["link"]. @@ -159,7 +159,7 @@ void Drive::DetectChanges() { Log( "Detecting changes from last sync", log::info ) ; Feed changes ; -// feed.EnableLog( "/tmp/changes", ".xml" ) ; + feed.EnableLog( "/tmp/changes", ".xml" ) ; feed.Start( &http, m_http_hdr, ChangesFeed(prev_stamp+1) ) ; std::for_each( diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed.cc index 72bfa9ac..a0fe9568 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed.cc @@ -69,8 +69,9 @@ void Feed::Start( http::Agent *http, const http::Header& auth, const std::string if ( m_log.get() != 0 ) log.Reset( - (boost::format( "%1%-%2%." ) % m_log->prefix % m_log->sequence++).str(), - m_log->suffix, &xrsp ) ; + m_log->prefix, + (boost::format( "-#%1%%2%" ) % m_log->sequence++ % m_log->suffix ).str(), + &xrsp ) ; http->Get( url, &log, auth ) ; diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index e253cac6..fb8d2315 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -260,8 +260,8 @@ void Resource::FromLocal( const DateTime& last_sync ) // in FromRemote() else { -// Trace( "file %1% mtime %2%, last_sync %3%", path, m_mtime, last_sync ) ; m_state = ( m_mtime > last_sync ? local_new : remote_deleted ) ; +// Trace( "file %1% mtime %2%, last_sync %3%, state (%4%)", path, m_mtime, last_sync, m_state ) ; } m_name = Path2Str( path.filename() ) ; @@ -567,7 +567,12 @@ bool Resource::Create( http::Agent* http, const http::Header& auth, DateTime& sy } } -bool Resource::Upload( http::Agent* http, const std::string& link, const http::Header& auth, bool post, DateTime& sync_time ) +bool Resource::Upload( + http::Agent* http, + const std::string& link, + const http::Header& auth, + bool post, + DateTime& sync_time ) { assert( http != 0 ) ; diff --git a/libgrive/src/http/ResponseLog.cc b/libgrive/src/http/ResponseLog.cc index 8de81fe4..3384bf7e 100644 --- a/libgrive/src/http/ResponseLog.cc +++ b/libgrive/src/http/ResponseLog.cc @@ -19,6 +19,7 @@ #include "ResponseLog.hh" +#include "util/log/Log.hh" #include "util/DateTime.hh" #include @@ -30,10 +31,9 @@ ResponseLog::ResponseLog( const std::string& suffix, Receivable *next ) : m_enabled ( true ), - m_log ( Filename(prefix, suffix).c_str() ), m_next ( next ) { - assert( m_next != 0 ) ; + Reset( prefix, suffix, next ) ; } ResponseLog::ResponseLog( Receivable *next ) : @@ -62,7 +62,7 @@ void ResponseLog::Clear() std::string ResponseLog::Filename( const std::string& prefix, const std::string& suffix ) { - return prefix + DateTime::Now().Format( "%H%M%S" ) + suffix ; + return prefix + DateTime::Now().Format( "%F.%H%M%S" ) + suffix ; } void ResponseLog::Enable( bool enable ) @@ -77,9 +77,23 @@ void ResponseLog::Reset( const std::string& prefix, const std::string& suffix, R if ( m_log.is_open() ) m_log.close() ; - m_log.open( Filename(prefix, suffix).c_str() ) ; + const std::string fname = Filename(prefix, suffix) ; + + // reset previous stream state. don't care if file can be opened + // successfully previously + m_log.clear() ; + + // re-open the file + m_log.open( fname.c_str() ) ; + if ( m_log ) + { + Trace( "logging HTTP response: %1%", fname ) ; + m_enabled = true ; + } + else + Trace( "cannot open log file %1%", fname ) ; + m_next = next ; - m_enabled = true ; } }} // end of namespace diff --git a/libgrive/src/util/log/Log.hh b/libgrive/src/util/log/Log.hh index 8bdd6bbb..0d43adbb 100644 --- a/libgrive/src/util/log/Log.hh +++ b/libgrive/src/util/log/Log.hh @@ -135,4 +135,10 @@ void Trace( const std::string& fmt, const P1& p1, const P2& p2, const P3& p3 ) LogBase::Inst()->Log( log::Fmt(fmt) % p1 % p2 % p3, log::debug ) ; } +template +void Trace( const std::string& fmt, const P1& p1, const P2& p2, const P3& p3, const P4& p4 ) +{ + LogBase::Inst()->Log( log::Fmt(fmt) % p1 % p2 % p3 % p4, log::debug ) ; +} + } // end of namespace From 94383e0030d8d0e3b06e90e52eb443d4aa99b488 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 16 Jul 2012 23:37:50 +0800 Subject: [PATCH 008/166] ignoring files with missing MD5 checksum in remote feeds --- libgrive/src/drive/Resource.cc | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index fb8d2315..5e7529d8 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -209,6 +209,14 @@ void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_sync ) } } + // remote checksum unknown, assume the file is not changed in remote + else if ( remote.MD5().empty() ) + { + Log( "file %1% has unknown checksum in remote. assuned in sync", + Path(), log::verbose ) ; + m_state = sync ; + } + // if checksum is equal, no need to compare the mtime else if ( remote.MD5() == m_md5 ) { @@ -259,10 +267,7 @@ void Resource::FromLocal( const DateTime& last_sync ) // remote_deleted first, it will be updated to sync/remote_changed // in FromRemote() else - { m_state = ( m_mtime > last_sync ? local_new : remote_deleted ) ; -// Trace( "file %1% mtime %2%, last_sync %3%, state (%4%)", path, m_mtime, last_sync, m_state ) ; - } m_name = Path2Str( path.filename() ) ; m_kind = fs::is_directory(path) ? "folder" : "file" ; From 75c67dad8d9861f5c3f64a9260af4f1692306a7a Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Tue, 17 Jul 2012 01:08:34 +0800 Subject: [PATCH 009/166] added class & functions for memory mapped files --- libgrive/src/drive/Resource.cc | 6 ++- libgrive/src/util/MemMap.cc | 46 +++++++++++++++++++ libgrive/src/util/MemMap.hh | 47 +++++++++++++++++++ libgrive/src/util/StdioFile.cc | 83 ++++++++++++++++++++++++++++++++-- libgrive/src/util/StdioFile.hh | 9 +++- libgrive/src/util/Types.hh | 31 +++++++++++++ 6 files changed, 214 insertions(+), 8 deletions(-) create mode 100644 libgrive/src/util/MemMap.cc create mode 100644 libgrive/src/util/MemMap.hh create mode 100644 libgrive/src/util/Types.hh diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 5e7529d8..0f056fc9 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -589,9 +589,11 @@ bool Resource::Upload( std::size_t count = 0 ; while ( (count = file.Read( buf, sizeof(buf) )) > 0 ) data.append( buf, count ) ; - + + assert( file.Size() == data.size() ) ; + std::ostringstream xcontent_len ; - xcontent_len << "X-Upload-Content-Length: " << data.size() ; + xcontent_len << "X-Upload-Content-Length: " << file.Size() ; http::Header hdr( auth ) ; hdr.Add( "Content-Type: application/atom+xml" ) ; diff --git a/libgrive/src/util/MemMap.cc b/libgrive/src/util/MemMap.cc new file mode 100644 index 00000000..6f8b4c77 --- /dev/null +++ b/libgrive/src/util/MemMap.cc @@ -0,0 +1,46 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "MemMap.hh" +#include "StdioFile.hh" + +namespace gr { + +MemMap::MemMap( StdioFile& file, off_t offset, std::size_t length ) : + m_addr ( file.Map( offset, length ) ), + m_length( length ) +{ +} + +MemMap::~MemMap() +{ + StdioFile::UnMap( m_addr, m_length ) ; +} + +void* MemMap::Addr() const +{ + return m_addr ; +} + +std::size_t MemMap::Length() const +{ + return m_length ; +} + +} // end of namespace diff --git a/libgrive/src/util/MemMap.hh b/libgrive/src/util/MemMap.hh new file mode 100644 index 00000000..91a30335 --- /dev/null +++ b/libgrive/src/util/MemMap.hh @@ -0,0 +1,47 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "Exception.hh" +#include "Types.hh" +#include + +namespace gr { + +class StdioFile ; + +class MemMap +{ +public : + struct Error : virtual Exception {} ; + +public : + MemMap( StdioFile& file, off_t offset, std::size_t length ) ; + ~MemMap() ; + + void* Addr() const ; + std::size_t Length() const ; + +private : + void *m_addr ; + std::size_t m_length ; +} ; + +} // end of namespace diff --git a/libgrive/src/util/StdioFile.cc b/libgrive/src/util/StdioFile.cc index ae0d41b5..435aea90 100644 --- a/libgrive/src/util/StdioFile.cc +++ b/libgrive/src/util/StdioFile.cc @@ -30,10 +30,47 @@ #include #include +#include #include #include #include +// local functions +namespace { + +off_t LSeek( int fd, off_t offset, int whence ) +{ + assert( fd >= 0 ) ; + + off_t r = ::lseek( fd, offset, whence ) ; + if ( r == static_cast(-1) ) + { + BOOST_THROW_EXCEPTION( + gr::StdioFile::Error() + << boost::errinfo_api_function("lseek") + << boost::errinfo_errno(errno) + ) ; + } + + return r ; +} + +struct stat FStat( int fd ) +{ + struct stat s = {} ; + if ( ::fstat( fd, &s ) != 0 ) + { + BOOST_THROW_EXCEPTION( + gr::StdioFile::Error() + << boost::errinfo_api_function("fstat") + << boost::errinfo_errno(errno) + ) ; + } + return s ; +} + +} // end of local functions + namespace gr { StdioFile::StdioFile( ) : m_fd( -1 ) @@ -127,16 +164,26 @@ std::size_t StdioFile::Write( const void *ptr, std::size_t size ) return count ; } -long StdioFile::Seek( long offset, int whence ) +off_t StdioFile::Seek( off_t offset, int whence ) +{ + assert( IsOpened() ) ; + return LSeek( m_fd, offset, whence ) ; +} + +off_t StdioFile::Tell() const { assert( IsOpened() ) ; - return ::lseek( m_fd, offset, whence ) ; + return LSeek( m_fd, 0, SEEK_CUR ) ; } -long StdioFile::Tell() const +u64_t StdioFile::Size() const { assert( IsOpened() ) ; - return ::lseek( m_fd, 0, SEEK_CUR ) ; + + struct stat s = FStat(m_fd) ; + + assert( s.st_size >= 0 ) ; + return static_cast( s.st_size ) ; } void StdioFile::Chmod( int mode ) @@ -153,4 +200,32 @@ void StdioFile::Chmod( int mode ) } } +void* StdioFile::Map( off_t offset, std::size_t length ) +{ + assert( IsOpened() ) ; + + void *addr = ::mmap( 0, length, PROT_READ, MAP_PRIVATE, m_fd, offset ) ; + if ( addr == reinterpret_cast( -1 ) ) + { + BOOST_THROW_EXCEPTION( + Error() + << boost::errinfo_api_function("mmap") + << boost::errinfo_errno(errno) + ) ; + } + return addr ; +} + +void StdioFile::UnMap( void *addr, std::size_t length ) +{ + if ( ::munmap( addr, length ) != 0 ) + { + BOOST_THROW_EXCEPTION( + Error() + << boost::errinfo_api_function("munmap") + << boost::errinfo_errno(errno) + ) ; + } +} + } // end of namespace diff --git a/libgrive/src/util/StdioFile.hh b/libgrive/src/util/StdioFile.hh index 30de2fb2..21443b6f 100644 --- a/libgrive/src/util/StdioFile.hh +++ b/libgrive/src/util/StdioFile.hh @@ -21,6 +21,7 @@ #include "Exception.hh" #include "FileSystem.hh" +#include "Types.hh" #include @@ -45,11 +46,15 @@ public : std::size_t Read( void *ptr, std::size_t size ) ; std::size_t Write( const void *ptr, std::size_t size ) ; - long Seek( long offset, int whence ) ; - long Tell() const ; + off_t Seek( off_t offset, int whence ) ; + off_t Tell() const ; + u64_t Size() const ; void Chmod( int mode ) ; + void* Map( off_t offset, std::size_t length ) ; + static void UnMap( void *addr, std::size_t length ) ; + private : void Open( const fs::path& path, int flags, int mode ) ; diff --git a/libgrive/src/util/Types.hh b/libgrive/src/util/Types.hh new file mode 100644 index 00000000..a981c6d3 --- /dev/null +++ b/libgrive/src/util/Types.hh @@ -0,0 +1,31 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include + +// import types into the Grive namespace +namespace gr +{ + using ::off_t ; + + // should use boost/cstdint + typedef unsigned long long u64_t ; +} From ce245576b51f9d88adbbdff966676787fd13f0a2 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Tue, 17 Jul 2012 01:58:16 +0800 Subject: [PATCH 010/166] use StdioFile for uploading (#88) --- libgrive/src/drive/Resource.cc | 12 +--------- libgrive/src/http/Agent.hh | 12 +++++++++- libgrive/src/http/CurlAgent.cc | 43 ++++++++++++++++++++++++++++++++-- libgrive/src/http/CurlAgent.hh | 6 +++++ 4 files changed, 59 insertions(+), 14 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 0f056fc9..b7e9ffa7 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -582,16 +582,6 @@ bool Resource::Upload( assert( http != 0 ) ; StdioFile file( Path() ) ; - - // TODO: upload in chunks - std::string data ; - char buf[4096] ; - std::size_t count = 0 ; - while ( (count = file.Read( buf, sizeof(buf) )) > 0 ) - data.append( buf, count ) ; - - assert( file.Size() == data.size() ) ; - std::ostringstream xcontent_len ; xcontent_len << "X-Upload-Content-Length: " << file.Size() ; @@ -621,7 +611,7 @@ bool Resource::Upload( std::string uplink = http->RedirLocation() ; http::XmlResponse xml ; - http->Put( uplink, data, &xml, uphdr ) ; + http->Put( uplink, file, &xml, uphdr ) ; AssignIDs( Entry( xml.Response() ) ) ; sync_time = std::max(Entry(xml.Response()).MTime(), sync_time); diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index b7b4779c..0baa0320 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -21,7 +21,11 @@ #include -namespace gr { namespace http { +namespace gr { + +class StdioFile ; + +namespace http { class Header ; class Receivable ; @@ -35,6 +39,12 @@ public : Receivable *dest, const Header& hdr ) = 0 ; + virtual long Put( + const std::string& url, + StdioFile& file, + Receivable *dest, + const Header& hdr ) = 0 ; + virtual long Get( const std::string& url, Receivable *dest, diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index f61f3cb2..ef70d693 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -25,6 +25,7 @@ #include "Receivable.hh" #include "util/log/Log.hh" +#include "util/StdioFile.hh" #include @@ -34,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -43,8 +45,9 @@ namespace { using namespace gr::http ; +using namespace gr ; -size_t ReadCallback( void *ptr, std::size_t size, std::size_t nmemb, std::string *data ) +size_t ReadStringCallback( void *ptr, std::size_t size, std::size_t nmemb, std::string *data ) { assert( ptr != 0 ) ; assert( data != 0 ) ; @@ -59,6 +62,22 @@ size_t ReadCallback( void *ptr, std::size_t size, std::size_t nmemb, std::string return count ; } +size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, StdioFile *file ) +{ + assert( ptr != 0 ) ; + assert( file != 0 ) ; + + u64_t count = std::min( + static_cast(size * nmemb), + static_cast(file->Size() - file->Tell()) ) ; + assert( count <= std::numeric_limits::max() ) ; + + if ( count > 0 ) + file->Read( ptr, static_cast(count) ) ; + + return count ; +} + } // end of local namespace namespace gr { namespace http { @@ -167,13 +186,33 @@ long CurlAgent::Put( // set common options ::curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L ) ; - ::curl_easy_setopt(curl, CURLOPT_READFUNCTION, &ReadCallback ) ; + ::curl_easy_setopt(curl, CURLOPT_READFUNCTION, &ReadStringCallback ) ; ::curl_easy_setopt(curl, CURLOPT_READDATA , &put_data ) ; ::curl_easy_setopt(curl, CURLOPT_INFILESIZE, put_data.size() ) ; return ExecCurl( url, dest, hdr ) ; } +long CurlAgent::Put( + const std::string& url, + StdioFile& file, + Receivable *dest, + const Header& hdr ) +{ + Trace("HTTP PUT \"%1%\"", url ) ; + + Init() ; + CURL *curl = m_pimpl->curl ; + + // set common options + ::curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L ) ; + ::curl_easy_setopt(curl, CURLOPT_READFUNCTION, &ReadFileCallback ) ; + ::curl_easy_setopt(curl, CURLOPT_READDATA , &file ) ; + ::curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, static_cast(file.Size()) ) ; + + return ExecCurl( url, dest, hdr ) ; +} + long CurlAgent::Get( const std::string& url, Receivable *dest, diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index b1756384..9db1d23b 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -44,6 +44,12 @@ public : const std::string& data, Receivable *dest, const Header& hdr ) ; + + long Put( + const std::string& url, + StdioFile& file, + Receivable *dest, + const Header& hdr ) ; long Get( const std::string& url, From 7cc4984932dad019a2ff11f8b32a1745445c9850 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Tue, 17 Jul 2012 01:50:41 +0800 Subject: [PATCH 011/166] use mmap for checksum --- libgrive/src/drive/Drive.cc | 4 ++-- libgrive/src/util/Crypt.cc | 15 +++++++++------ 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index f83bdd64..f0d454f1 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -140,7 +140,7 @@ void Drive::DetectChanges() Log( "Reading remote server file list", log::info ) ; Feed feed ; - feed.EnableLog( "/tmp/file", ".xml" ) ; +// feed.EnableLog( "/tmp/file", ".xml" ) ; feed.Start( &http, m_http_hdr, feed_base + "?showfolders=true&showroot=true" ) ; m_resume_link = feed.Root()["link"]. @@ -159,7 +159,7 @@ void Drive::DetectChanges() { Log( "Detecting changes from last sync", log::info ) ; Feed changes ; - feed.EnableLog( "/tmp/changes", ".xml" ) ; +// feed.EnableLog( "/tmp/changes", ".xml" ) ; feed.Start( &http, m_http_hdr, ChangesFeed(prev_stamp+1) ) ; std::for_each( diff --git a/libgrive/src/util/Crypt.cc b/libgrive/src/util/Crypt.cc index 6b9ca329..953767d3 100644 --- a/libgrive/src/util/Crypt.cc +++ b/libgrive/src/util/Crypt.cc @@ -21,6 +21,7 @@ #include "StdioFile.hh" #include "Exception.hh" +#include "MemMap.hh" #include #include @@ -31,7 +32,8 @@ namespace gr { namespace crypt { -const std::size_t read_size = 8 * 1024 ; +// map 4MB of data at a time +const u64_t read_size = 1024 * 4096 ; struct MD5::Impl { @@ -85,13 +87,14 @@ std::string MD5::Get( const fs::path& file ) std::string MD5::Get( StdioFile& file ) { - char buf[read_size] ; - MD5 crypt ; - std::size_t count = 0 ; - while ( (count = file.Read( buf, sizeof(buf) )) > 0 ) - crypt.Write( buf, count ) ; + u64_t size = file.Size() ; + for ( u64_t i = 0 ; i < size ; i += read_size ) + { + MemMap map( file, i, static_cast(std::min(read_size, size-i)) ) ; + crypt.Write( map.Addr(), map.Length() ) ; + } return crypt.Get() ; } From 56de93c8c794e0e7f4d3acfc5f6817fe5f011470 Mon Sep 17 00:00:00 2001 From: Jan Winkler Date: Thu, 19 Jul 2012 16:51:03 +0200 Subject: [PATCH 012/166] Commented out iberty_library stuff The iberty_library could not be found anywhere in my repos or on my system. Compiles and runs fine without. --- CMakeLists.txt | 3 +++ libgrive/CMakeLists.txt | 9 +++++---- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index ca39cb19..c2137abf 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3,6 +3,9 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! set( GRIVE_VERSION "0.2.0" ) +# Actually find the boost libraries +find_package( Boost ) + # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) add_definitions( -D_FILE_OFFSET_BITS=64 ) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index a19fc992..748f3888 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -27,9 +27,10 @@ if ( BFD_FOUND ) endif ( BFD_FOUND ) -if ( IBERTY_FOUND ) - set( OPT_LIBS ${OPT_LIBS} ${IBERTY_LIBRARY} ) -endif ( IBERTY_FOUND ) +# Not found on my system or in any standard repos. +#if ( IBERTY_FOUND ) +# set( OPT_LIBS ${OPT_LIBS} ${IBERTY_LIBRARY} ) +#endif ( IBERTY_FOUND ) if ( ZLIB_FOUND ) set( OPT_LIBS ${OPT_LIBS} ${ZLIB_LIBRARIES} ) @@ -81,7 +82,7 @@ target_link_libraries( grive ${LIBGCRYPT_LIBRARIES} ${GDBM_LIBRARIES} ${Boost_LIBRARIES} - ${IBERTY_LIBRARY} + #${IBERTY_LIBRARY} ${EXPAT_LIBRARY} ${OPT_LIBS} ) From 2f9416859907520ad2b589d3ad02b2cdb5a7b714 Mon Sep 17 00:00:00 2001 From: Jan Winkler Date: Thu, 19 Jul 2012 17:19:40 +0200 Subject: [PATCH 013/166] Set up an else-clause for IBERTY_LIBRARY and removed unnecessary BOOST-find In ./CMakeLists.txt the added find_package( Boost ) was removed due to being unnecessary. In libgrive/CMakeLists.txt the IBERTY_LIBRARY is now set to an empty string of it was not found (otherwise the linker breaks during `cmake'). --- CMakeLists.txt | 3 --- libgrive/CMakeLists.txt | 11 ++++++----- 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index c2137abf..ca39cb19 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3,9 +3,6 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! set( GRIVE_VERSION "0.2.0" ) -# Actually find the boost libraries -find_package( Boost ) - # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) add_definitions( -D_FILE_OFFSET_BITS=64 ) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 748f3888..5008a3ca 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -27,10 +27,11 @@ if ( BFD_FOUND ) endif ( BFD_FOUND ) -# Not found on my system or in any standard repos. -#if ( IBERTY_FOUND ) -# set( OPT_LIBS ${OPT_LIBS} ${IBERTY_LIBRARY} ) -#endif ( IBERTY_FOUND ) +if ( IBERTY_FOUND ) + set( OPT_LIBS ${OPT_LIBS} ${IBERTY_LIBRARY} ) +else ( IBERTY_FOUND ) + set( IBERTY_LIBRARY "" ) +endif ( IBERTY_FOUND ) if ( ZLIB_FOUND ) set( OPT_LIBS ${OPT_LIBS} ${ZLIB_LIBRARIES} ) @@ -82,7 +83,7 @@ target_link_libraries( grive ${LIBGCRYPT_LIBRARIES} ${GDBM_LIBRARIES} ${Boost_LIBRARIES} - #${IBERTY_LIBRARY} + ${IBERTY_LIBRARY} ${EXPAT_LIBRARY} ${OPT_LIBS} ) From 57c431dd83574486a762479b9ff26a56a49f8787 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 19 Jul 2012 23:48:01 +0800 Subject: [PATCH 014/166] added logs for tracing last sync time --- libgrive/src/drive/State.cc | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 280c071f..f024818c 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -273,9 +273,14 @@ void State::Sync( http::Agent *http, const http::Header& auth ) // need to check if this introduces a new problem DateTime last_sync_time = m_last_sync; m_res.Root()->Sync( http, auth, last_sync_time ) ; - if (last_sync_time == m_last_sync) { + if ( last_sync_time == m_last_sync ) + { + Trace( "nothing changed? %1%", m_last_sync ) ; m_last_sync = DateTime::Now(); - } else { + } + else + { + Trace( "updating last sync? %1%", last_sync_time ) ; m_last_sync = last_sync_time; } } From 28e8012ca52359c73fefab4bbf4f81030cadafbb Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Fri, 20 Jul 2012 00:39:36 +0800 Subject: [PATCH 015/166] added AuthAgent, a wrapper around http agent --- grive/src/main.cc | 5 +- libgrive/src/drive/Drive.cc | 42 ++++++------ libgrive/src/drive/Drive.hh | 12 ++-- libgrive/src/drive/Resource.cc | 39 ++++++----- libgrive/src/drive/Resource.hh | 15 ++--- libgrive/src/drive/State.cc | 5 +- libgrive/src/drive/State.hh | 3 +- libgrive/src/http/Header.cc | 7 ++ libgrive/src/http/Header.hh | 1 + libgrive/src/protocol/AuthAgent.cc | 103 +++++++++++++++++++++++++++++ libgrive/src/protocol/AuthAgent.hh | 81 +++++++++++++++++++++++ 11 files changed, 250 insertions(+), 63 deletions(-) create mode 100644 libgrive/src/protocol/AuthAgent.cc create mode 100644 libgrive/src/protocol/AuthAgent.hh diff --git a/grive/src/main.cc b/grive/src/main.cc index d51e0d92..47947f39 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -21,6 +21,8 @@ #include "drive/Drive.hh" +#include "http/CurlAgent.hh" +#include "protocol/AuthAgent.hh" #include "protocol/OAuth2.hh" #include "protocol/Json.hh" @@ -172,7 +174,8 @@ int Main( int argc, char **argv ) } OAuth2 token( refresh_token, client_id, client_secret ) ; - Drive drive( token, options ) ; + AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; + Drive drive( &agent, options ) ; drive.DetectChanges() ; if ( vm.count( "dry-run" ) == 0 ) diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index f0d454f1..daf0a725 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -23,11 +23,11 @@ #include "Entry.hh" #include "Feed.hh" -#include "http/CurlAgent.hh" +#include "http/Agent.hh" #include "http/ResponseLog.hh" #include "http/XmlResponse.hh" #include "protocol/Json.hh" -#include "protocol/OAuth2.hh" +// #include "protocol/OAuth2.hh" #include "util/Destroy.hh" #include "util/log/Log.hh" #include "xml/Node.hh" @@ -53,12 +53,11 @@ namespace const std::string state_file = ".grive_state" ; } -Drive::Drive( OAuth2& auth, const Json& options ) : - m_auth( auth ), +Drive::Drive( http::Agent *http, const Json& options ) : + m_http( http ), m_state( state_file, options ) { - m_http_hdr.Add( "Authorization: Bearer " + m_auth.AccessToken() ) ; - m_http_hdr.Add( "GData-Version: 3.0" ) ; + assert( m_http != 0 ) ; } void Drive::FromRemote( const Entry& entry ) @@ -93,14 +92,14 @@ void Drive::SaveState() m_state.Write( state_file ) ; } -void Drive::SyncFolders( http::Agent *http ) +void Drive::SyncFolders( ) { - assert( http != 0 ) ; + assert( m_http != 0 ) ; Log( "Synchronizing folders", log::info ) ; http::XmlResponse xml ; - http->Get( feed_base + "/-/folder?max-results=50&showroot=true", &xml, m_http_hdr ) ; + m_http->Get( feed_base + "/-/folder?max-results=50&showroot=true", &xml, http::Header() ) ; Feed feed( xml.Response() ) ; do @@ -121,7 +120,7 @@ void Drive::SyncFolders( http::Agent *http ) m_state.FromRemote( e ) ; } } - } while ( feed.GetNext( http, m_http_hdr ) ) ; + } while ( feed.GetNext( m_http, http::Header() ) ) ; m_state.ResolveEntry() ; } @@ -131,17 +130,15 @@ void Drive::DetectChanges() Log( "Reading local directories", log::info ) ; m_state.FromLocal( "." ) ; - http::CurlAgent http ; - long prev_stamp = m_state.ChangeStamp() ; Trace( "previous change stamp is %1%", prev_stamp ) ; - SyncFolders( &http ) ; + SyncFolders( ) ; Log( "Reading remote server file list", log::info ) ; Feed feed ; // feed.EnableLog( "/tmp/file", ".xml" ) ; - feed.Start( &http, m_http_hdr, feed_base + "?showfolders=true&showroot=true" ) ; + feed.Start( m_http, http::Header(), feed_base + "?showfolders=true&showroot=true" ) ; m_resume_link = feed.Root()["link"]. Find( "@rel", "http://schemas.google.com/g/2005#resumable-create-media" )["@href"] ; @@ -152,7 +149,7 @@ void Drive::DetectChanges() feed.begin(), feed.end(), boost::bind( &Drive::FromRemote, this, _1 ) ) ; - } while ( feed.GetNext( &http, m_http_hdr ) ) ; + } while ( feed.GetNext( m_http, http::Header() ) ) ; // pull the changes feed if ( prev_stamp != -1 ) @@ -160,7 +157,7 @@ void Drive::DetectChanges() Log( "Detecting changes from last sync", log::info ) ; Feed changes ; // feed.EnableLog( "/tmp/changes", ".xml" ) ; - feed.Start( &http, m_http_hdr, ChangesFeed(prev_stamp+1) ) ; + feed.Start( m_http, http::Header(), ChangesFeed(prev_stamp+1) ) ; std::for_each( changes.begin(), changes.end(), @@ -171,25 +168,24 @@ void Drive::DetectChanges() void Drive::Update() { Log( "Synchronizing files", log::info ) ; - http::CurlAgent http ; - m_state.Sync( &http, m_http_hdr ) ; + m_state.Sync( m_http ) ; - UpdateChangeStamp( &http ) ; + UpdateChangeStamp( ) ; } void Drive::DryRun() { Log( "Synchronizing files (dry-run)", log::info ) ; - m_state.Sync( 0, m_http_hdr ) ; + m_state.Sync( 0 ) ; } -void Drive::UpdateChangeStamp( http::Agent *http ) +void Drive::UpdateChangeStamp( ) { - assert( http != 0 ) ; + assert( m_http != 0 ) ; // get changed feed http::XmlResponse xrsp ; - http->Get( ChangesFeed(m_state.ChangeStamp()+1), &xrsp, m_http_hdr ) ; + m_http->Get( ChangesFeed(m_state.ChangeStamp()+1), &xrsp, http::Header() ) ; // we should go through the changes to see if it was really Grive to made that change // maybe by recording the updated timestamp and compare it? diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 9a6227c2..3641500c 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -35,13 +35,12 @@ namespace http } class Entry ; -class OAuth2 ; class Json ; class Drive { public : - Drive( OAuth2& auth, const Json& options ) ; + Drive( http::Agent *http, const Json& options ) ; void DetectChanges() ; void Update() ; @@ -51,15 +50,16 @@ public : struct Error : virtual Exception {} ; private : - void SyncFolders( http::Agent *http ) ; + void SyncFolders( ) ; void file(); void FromRemote( const Entry& entry ) ; void FromChange( const Entry& entry ) ; - void UpdateChangeStamp( http::Agent *http ) ; + void UpdateChangeStamp( ) ; private : - OAuth2& m_auth ; - http::Header m_http_hdr ; + http::Agent *m_http ; +// OAuth2& m_auth ; +// http::Header m_http_hdr ; std::string m_resume_link ; State m_state ; diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index b7e9ffa7..268c92f0 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -365,20 +365,20 @@ Resource* Resource::FindChild( const std::string& name ) } // try to change the state to "sync" -void Resource::Sync( http::Agent *http, const http::Header& auth, DateTime& sync_time ) +void Resource::Sync( http::Agent *http, DateTime& sync_time ) { assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced - SyncSelf( http, auth, sync_time ) ; + SyncSelf( http, sync_time ) ; // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) std::for_each( m_child.begin(), m_child.end(), - boost::bind( &Resource::Sync, _1, http, auth, boost::ref(sync_time) ) ) ; + boost::bind( &Resource::Sync, _1, http, boost::ref(sync_time) ) ) ; } -void Resource::SyncSelf( http::Agent* http, const http::Header& auth, DateTime& sync_time ) +void Resource::SyncSelf( http::Agent* http, DateTime& sync_time ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync assert( IsRoot() || http == 0 || fs::is_directory( m_parent->Path() ) ) ; @@ -392,19 +392,19 @@ void Resource::SyncSelf( http::Agent* http, const http::Header& auth, DateTime& case local_new : Log( "sync %1% doesn't exist in server, uploading", path, log::info ) ; - if ( http != 0 && Create( http, auth, sync_time ) ) + if ( http != 0 && Create( http, sync_time ) ) m_state = sync ; break ; case local_deleted : Log( "sync %1% deleted in local. deleting remote", path, log::info ) ; if ( http != 0 ) - DeleteRemote( http, auth ) ; + DeleteRemote( http ) ; break ; case local_changed : Log( "sync %1% changed in local. uploading", path, log::info ) ; - if ( http != 0 && EditContent( http, auth, sync_time ) ) + if ( http != 0 && EditContent( http, sync_time ) ) m_state = sync ; break ; @@ -415,7 +415,7 @@ void Resource::SyncSelf( http::Agent* http, const http::Header& auth, DateTime& if ( IsFolder() ) fs::create_directories( path ) ; else - Download( http, path, auth ) ; + Download( http, path ) ; m_state = sync ; } @@ -426,7 +426,7 @@ void Resource::SyncSelf( http::Agent* http, const http::Header& auth, DateTime& Log( "sync %1% changed in remote. downloading", path, log::info ) ; if ( http != 0 ) { - Download( http, path, auth ) ; + Download( http, path ) ; m_state = sync ; } break ; @@ -474,14 +474,14 @@ void Resource::DeleteLocal() } } -void Resource::DeleteRemote( http::Agent *http, const http::Header& auth ) +void Resource::DeleteRemote( http::Agent *http ) { assert( http != 0 ) ; http::StringResponse str ; try { - http::Header hdr( auth ) ; + http::Header hdr ; hdr.Add( "If-Match: " + m_etag ) ; // doesn't know why, but an update before deleting seems to work always @@ -502,12 +502,12 @@ void Resource::DeleteRemote( http::Agent *http, const http::Header& auth ) } -void Resource::Download( http::Agent* http, const fs::path& file, const http::Header& auth ) const +void Resource::Download( http::Agent* http, const fs::path& file ) const { assert( http != 0 ) ; http::Download dl( file.string(), http::Download::NoChecksum() ) ; - long r = http->Get( m_content, &dl, auth ) ; + long r = http->Get( m_content, &dl, http::Header() ) ; if ( r <= 400 ) { if ( m_mtime != DateTime() ) @@ -517,7 +517,7 @@ void Resource::Download( http::Agent* http, const fs::path& file, const http::He } } -bool Resource::EditContent( http::Agent* http, const http::Header& auth, DateTime& sync_time ) +bool Resource::EditContent( http::Agent* http, DateTime& sync_time ) { assert( http != 0 ) ; assert( m_parent != 0 ) ; @@ -530,10 +530,10 @@ bool Resource::EditContent( http::Agent* http, const http::Header& auth, DateTim return false ; } - return Upload( http, m_edit, auth, false, sync_time ) ; + return Upload( http, m_edit, false, sync_time ) ; } -bool Resource::Create( http::Agent* http, const http::Header& auth, DateTime& sync_time ) +bool Resource::Create( http::Agent* http, DateTime& sync_time ) { assert( http != 0 ) ; assert( m_parent != 0 ) ; @@ -551,7 +551,7 @@ bool Resource::Create( http::Agent* http, const http::Header& auth, DateTime& sy % xml::Escape(m_name) ).str() ; - http::Header hdr( auth ) ; + http::Header hdr ; hdr.Add( "Content-Type: application/atom+xml" ) ; http::XmlResponse xml ; @@ -563,7 +563,7 @@ bool Resource::Create( http::Agent* http, const http::Header& auth, DateTime& sy } else if ( !m_parent->m_create.empty() ) { - return Upload( http, m_parent->m_create + "?convert=false", auth, true, sync_time ) ; + return Upload( http, m_parent->m_create + "?convert=false", true, sync_time ) ; } else { @@ -575,7 +575,6 @@ bool Resource::Create( http::Agent* http, const http::Header& auth, DateTime& sy bool Resource::Upload( http::Agent* http, const std::string& link, - const http::Header& auth, bool post, DateTime& sync_time ) { @@ -585,7 +584,7 @@ bool Resource::Upload( std::ostringstream xcontent_len ; xcontent_len << "X-Upload-Content-Length: " << file.Size() ; - http::Header hdr( auth ) ; + http::Header hdr ; hdr.Add( "Content-Type: application/atom+xml" ) ; hdr.Add( "X-Upload-Content-Type: application/octet-stream" ) ; hdr.Add( xcontent_len.str() ) ; diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 8a76017c..75efdc40 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -32,7 +32,6 @@ namespace gr { namespace http { class Agent ; - class Header ; } class Entry ; @@ -77,7 +76,7 @@ public : void FromRemote( const Entry& remote, const DateTime& last_sync ) ; void FromLocal( const DateTime& last_sync ) ; - void Sync( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; + void Sync( http::Agent* http, DateTime& sync_time ) ; // children access iterator begin() const ; @@ -124,19 +123,19 @@ private : private : void SetState( State new_state ) ; - void Download( http::Agent* http, const fs::path& file, const http::Header& auth ) const ; - bool EditContent( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; - bool Create( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; - bool Upload( http::Agent* http, const std::string& link, const http::Header& auth, bool post, DateTime& sync_time ) ; + void Download( http::Agent* http, const fs::path& file ) const ; + bool EditContent( http::Agent* http, DateTime& sync_time ) ; + bool Create( http::Agent* http, DateTime& sync_time ) ; + bool Upload( http::Agent* http, const std::string& link, bool post, DateTime& sync_time ) ; void FromRemoteFolder( const Entry& remote, const DateTime& last_sync ) ; void FromRemoteFile( const Entry& remote, const DateTime& last_sync ) ; void DeleteLocal() ; - void DeleteRemote( http::Agent* http, const http::Header& auth ) ; + void DeleteRemote( http::Agent* http ) ; void AssignIDs( const Entry& remote ) ; - void SyncSelf( http::Agent* http, const http::Header& auth, DateTime& sync_time ) ; + void SyncSelf( http::Agent* http, DateTime& sync_time ) ; private : std::string m_name ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index f024818c..afab8420 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -24,7 +24,6 @@ #include "CommonUri.hh" #include "http/Agent.hh" -#include "http/Header.hh" #include "util/Crypt.hh" #include "util/log/Log.hh" #include "protocol/Json.hh" @@ -262,7 +261,7 @@ void State::Write( const fs::path& filename ) const fs << result ; } -void State::Sync( http::Agent *http, const http::Header& auth ) +void State::Sync( http::Agent *http ) { // set the last sync time from the time returned by the server for the last file synced // if the sync time hasn't changed (i.e. now files have been uploaded) @@ -272,7 +271,7 @@ void State::Sync( http::Agent *http, const http::Header& auth ) // TODO - WARNING - do we use the last sync time to compare to client file times // need to check if this introduces a new problem DateTime last_sync_time = m_last_sync; - m_res.Root()->Sync( http, auth, last_sync_time ) ; + m_res.Root()->Sync( http, last_sync_time ) ; if ( last_sync_time == m_last_sync ) { Trace( "nothing changed? %1%", m_last_sync ) ; diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index 563abe56..cd0590d4 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -31,7 +31,6 @@ namespace gr { namespace http { class Agent ; - class Header ; } class Json ; @@ -58,7 +57,7 @@ public : Resource* FindByID( const std::string& id ) ; Resource* Find( const fs::path& path ) ; - void Sync( http::Agent *http, const http::Header& auth ) ; + void Sync( http::Agent *http ) ; iterator begin() ; iterator end() ; diff --git a/libgrive/src/http/Header.cc b/libgrive/src/http/Header.cc index 36bd245a..9f0bdd31 100644 --- a/libgrive/src/http/Header.cc +++ b/libgrive/src/http/Header.cc @@ -50,4 +50,11 @@ std::ostream& operator<<( std::ostream& os, const Header& h ) return os ; } +Header operator+( const Header& header, const std::string& str ) +{ + Header h( header ) ; + h.Add( str ) ; + return h ; +} + } } // end of namespace diff --git a/libgrive/src/http/Header.hh b/libgrive/src/http/Header.hh index c581d7d1..2486cd0c 100644 --- a/libgrive/src/http/Header.hh +++ b/libgrive/src/http/Header.hh @@ -46,5 +46,6 @@ private : } ; std::ostream& operator<<( std::ostream& os, const Header& h ) ; +Header operator+( const Header& header, const std::string& str ) ; }} // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc new file mode 100644 index 00000000..06cacd21 --- /dev/null +++ b/libgrive/src/protocol/AuthAgent.cc @@ -0,0 +1,103 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "AuthAgent.hh" +#include "http/Header.hh" + +#include + +namespace gr { + +using namespace http ; + +AuthAgent::AuthAgent( const OAuth2& auth, std::auto_ptr real_agent ) : + m_auth ( auth ), + m_agent ( real_agent ) +{ + assert( m_agent.get() != 0 ) ; +} + +Header AuthAgent::AppendHeader( const Header& hdr ) const +{ + Header h(hdr) ; + h.Add( "Authorization: Bearer " + m_auth.AccessToken() ) ; + h.Add( "GData-Version: 3.0" ) ; + return h ; +} + +long AuthAgent::Put( + const std::string& url, + const std::string& data, + Receivable *dest, + const Header& hdr ) +{ + return m_agent->Put( url, data, dest, AppendHeader(hdr) ) ; +} + +long AuthAgent::Put( + const std::string& url, + StdioFile& file, + Receivable *dest, + const Header& hdr ) +{ + return m_agent->Put( url, file, dest, AppendHeader(hdr) ) ; +} + +long AuthAgent::Get( + const std::string& url, + Receivable *dest, + const Header& hdr ) +{ + return m_agent->Get( url, dest, AppendHeader(hdr) ) ; +} + +long AuthAgent::Post( + const std::string& url, + const std::string& data, + Receivable *dest, + const Header& hdr ) +{ + return m_agent->Post( url, data, dest, AppendHeader(hdr) ) ; +} + +long AuthAgent::Custom( + const std::string& method, + const std::string& url, + Receivable *dest, + const Header& hdr ) +{ + return m_agent->Custom( method, url, dest, AppendHeader(hdr) ) ; +} + +std::string AuthAgent::RedirLocation() const +{ + return m_agent->RedirLocation() ; +} + +std::string AuthAgent::Escape( const std::string& str ) +{ + return m_agent->Escape( str ) ; +} + +std::string AuthAgent::Unescape( const std::string& str ) +{ + return m_agent->Unescape( str ) ; +} + +} // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh new file mode 100644 index 00000000..d3fb0534 --- /dev/null +++ b/libgrive/src/protocol/AuthAgent.hh @@ -0,0 +1,81 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "http/Agent.hh" +#include "OAuth2.hh" + +#include + +namespace gr { + +/*! \brief An HTTP agent with support OAuth2 + + This is a HTTP agent that provide support for OAuth2. It will also perform retries on + certain HTTP errors. +*/ +class AuthAgent : public http::Agent +{ +public : + AuthAgent( const OAuth2& auth, std::auto_ptr real_agent ) ; + + long Put( + const std::string& url, + const std::string& data, + http::Receivable *dest, + const http::Header& hdr ) ; + + long Put( + const std::string& url, + StdioFile& file, + http::Receivable *dest, + const http::Header& hdr ) ; + + long Get( + const std::string& url, + http::Receivable *dest, + const http::Header& hdr ) ; + + long Post( + const std::string& url, + const std::string& data, + http::Receivable *dest, + const http::Header& hdr ) ; + + long Custom( + const std::string& method, + const std::string& url, + http::Receivable *dest, + const http::Header& hdr ) ; + + std::string RedirLocation() const ; + + std::string Escape( const std::string& str ) ; + std::string Unescape( const std::string& str ) ; + +private : + http::Header AppendHeader( const http::Header& hdr ) const ; + +private : + OAuth2 m_auth ; + const std::auto_ptr m_agent ; +} ; + +} // end of namespace From c6a1c47d4021a6c9a0c8820b0052f148474d6bec Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Fri, 20 Jul 2012 00:50:40 +0800 Subject: [PATCH 016/166] added retry on HTTP 500 & 503 (#82) --- libgrive/src/http/CurlAgent.cc | 2 +- libgrive/src/protocol/AuthAgent.cc | 47 ++++++++++++++++++++++++++---- libgrive/src/protocol/AuthAgent.hh | 1 + libgrive/src/util/OS.cc | 13 +++++++++ libgrive/src/util/OS.hh | 2 ++ 5 files changed, 59 insertions(+), 6 deletions(-) diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index ef70d693..85dfe38e 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -156,7 +156,7 @@ long CurlAgent::ExecCurl( Trace( "HTTP response %1%", http_code ) ; ::curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, 0 ) ; - if ( curl_code != CURLE_OK || http_code >= 400 ) + if ( curl_code != CURLE_OK || (http_code >= 400 && http_code < 500) ) { BOOST_THROW_EXCEPTION( Error() diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 06cacd21..b54f2324 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -18,7 +18,10 @@ */ #include "AuthAgent.hh" + #include "http/Header.hh" +#include "util/log/Log.hh" +#include "util/OS.hh" #include @@ -47,7 +50,11 @@ long AuthAgent::Put( Receivable *dest, const Header& hdr ) { - return m_agent->Put( url, data, dest, AppendHeader(hdr) ) ; + long response ; + while ( CheckRetry( + response = m_agent->Put(url, data, dest, AppendHeader(hdr)) ) ) ; + + return response ; } long AuthAgent::Put( @@ -56,7 +63,11 @@ long AuthAgent::Put( Receivable *dest, const Header& hdr ) { - return m_agent->Put( url, file, dest, AppendHeader(hdr) ) ; + long response ; + while ( CheckRetry( + response = m_agent->Put( url, file, dest, AppendHeader(hdr) ) ) ) ; + + return response ; } long AuthAgent::Get( @@ -64,7 +75,11 @@ long AuthAgent::Get( Receivable *dest, const Header& hdr ) { - return m_agent->Get( url, dest, AppendHeader(hdr) ) ; + long response ; + while ( CheckRetry( + response = m_agent->Get( url, dest, AppendHeader(hdr) ) ) ) ; + + return response ; } long AuthAgent::Post( @@ -73,7 +88,11 @@ long AuthAgent::Post( Receivable *dest, const Header& hdr ) { - return m_agent->Post( url, data, dest, AppendHeader(hdr) ) ; + long response ; + while ( CheckRetry( + response = m_agent->Post( url, data, dest, AppendHeader(hdr) ) ) ) ; + + return response ; } long AuthAgent::Custom( @@ -82,7 +101,11 @@ long AuthAgent::Custom( Receivable *dest, const Header& hdr ) { - return m_agent->Custom( method, url, dest, AppendHeader(hdr) ) ; + long response ; + while ( CheckRetry( + response = m_agent->Custom( method, url, dest, AppendHeader(hdr) ) ) ) ; + + return response ; } std::string AuthAgent::RedirLocation() const @@ -100,4 +123,18 @@ std::string AuthAgent::Unescape( const std::string& str ) return m_agent->Unescape( str ) ; } +bool AuthAgent::CheckRetry( long response ) +{ + if ( response == 500 || response == 503 ) + { + Log( "resquest failed due to temperory error: %1%. retrying in 5 seconds", + response, log::warning ) ; + + os::Sleep( 5 ) ; + return true ; + } + else + return false ; +} + } // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index d3fb0534..3b198c60 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -72,6 +72,7 @@ public : private : http::Header AppendHeader( const http::Header& hdr ) const ; + bool CheckRetry( long response ) ; private : OAuth2 m_auth ; diff --git a/libgrive/src/util/OS.cc b/libgrive/src/util/OS.cc index 0d51136b..5bf2f663 100644 --- a/libgrive/src/util/OS.cc +++ b/libgrive/src/util/OS.cc @@ -81,4 +81,17 @@ void SetFileTime( const std::string& filename, const DateTime& t ) ) ; } +void Sleep( unsigned int sec ) +{ + struct timespec ts = { sec, 0 } ; + + int result = 0 ; + do + { + struct timespec rem ; + nanosleep( &ts, &rem ) ; + ts = rem ; + } while ( result == -1 && errno == EINTR ) ; +} + } } // end of namespaces diff --git a/libgrive/src/util/OS.hh b/libgrive/src/util/OS.hh index 4a0350dc..31ab51de 100644 --- a/libgrive/src/util/OS.hh +++ b/libgrive/src/util/OS.hh @@ -38,6 +38,8 @@ namespace os void SetFileTime( const std::string& filename, const DateTime& t ) ; void SetFileTime( const fs::path& filename, const DateTime& t ) ; + + void Sleep( unsigned int sec ) ; } } // end of namespaces From 96227d38cfc76c68b68a0220316b308033c5ee24 Mon Sep 17 00:00:00 2001 From: justin Date: Thu, 19 Jul 2012 22:22:34 +0100 Subject: [PATCH 017/166] store modified time of file in m_mtime instead of passing value as param --- libgrive/src/drive/Resource.cc | 24 +++++++++++++----------- libgrive/src/drive/Resource.hh | 8 ++++---- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 268c92f0..c044abc8 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -370,7 +370,10 @@ void Resource::Sync( http::Agent *http, DateTime& sync_time ) assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced - SyncSelf( http, sync_time ) ; + SyncSelf( http ) ; + // we want the server sync time, so we will take the server time of the last file uploaded to store as the sync time + // m_mtime is updated to server modified time when the file is uploaded + sync_time = std::max(sync_time, m_mtime); // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) @@ -378,7 +381,7 @@ void Resource::Sync( http::Agent *http, DateTime& sync_time ) boost::bind( &Resource::Sync, _1, http, boost::ref(sync_time) ) ) ; } -void Resource::SyncSelf( http::Agent* http, DateTime& sync_time ) +void Resource::SyncSelf( http::Agent* http ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync assert( IsRoot() || http == 0 || fs::is_directory( m_parent->Path() ) ) ; @@ -392,7 +395,7 @@ void Resource::SyncSelf( http::Agent* http, DateTime& sync_time ) case local_new : Log( "sync %1% doesn't exist in server, uploading", path, log::info ) ; - if ( http != 0 && Create( http, sync_time ) ) + if ( http != 0 && Create( http ) ) m_state = sync ; break ; @@ -404,7 +407,7 @@ void Resource::SyncSelf( http::Agent* http, DateTime& sync_time ) case local_changed : Log( "sync %1% changed in local. uploading", path, log::info ) ; - if ( http != 0 && EditContent( http, sync_time ) ) + if ( http != 0 && EditContent( http ) ) m_state = sync ; break ; @@ -517,7 +520,7 @@ void Resource::Download( http::Agent* http, const fs::path& file ) const } } -bool Resource::EditContent( http::Agent* http, DateTime& sync_time ) +bool Resource::EditContent( http::Agent* http ) { assert( http != 0 ) ; assert( m_parent != 0 ) ; @@ -530,10 +533,10 @@ bool Resource::EditContent( http::Agent* http, DateTime& sync_time ) return false ; } - return Upload( http, m_edit, false, sync_time ) ; + return Upload( http, m_edit, false ) ; } -bool Resource::Create( http::Agent* http, DateTime& sync_time ) +bool Resource::Create( http::Agent* http) { assert( http != 0 ) ; assert( m_parent != 0 ) ; @@ -563,7 +566,7 @@ bool Resource::Create( http::Agent* http, DateTime& sync_time ) } else if ( !m_parent->m_create.empty() ) { - return Upload( http, m_parent->m_create + "?convert=false", true, sync_time ) ; + return Upload( http, m_parent->m_create + "?convert=false", true ) ; } else { @@ -575,8 +578,7 @@ bool Resource::Create( http::Agent* http, DateTime& sync_time ) bool Resource::Upload( http::Agent* http, const std::string& link, - bool post, - DateTime& sync_time ) + bool post) { assert( http != 0 ) ; @@ -612,7 +614,7 @@ bool Resource::Upload( http->Put( uplink, file, &xml, uphdr ) ; AssignIDs( Entry( xml.Response() ) ) ; - sync_time = std::max(Entry(xml.Response()).MTime(), sync_time); + m_mtime = Entry(xml.Response()).MTime(); return true ; } diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 75efdc40..b0cb2b8e 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -124,9 +124,9 @@ private : void SetState( State new_state ) ; void Download( http::Agent* http, const fs::path& file ) const ; - bool EditContent( http::Agent* http, DateTime& sync_time ) ; - bool Create( http::Agent* http, DateTime& sync_time ) ; - bool Upload( http::Agent* http, const std::string& link, bool post, DateTime& sync_time ) ; + bool EditContent( http::Agent* http ) ; + bool Create( http::Agent* http ) ; + bool Upload( http::Agent* http, const std::string& link, bool post ) ; void FromRemoteFolder( const Entry& remote, const DateTime& last_sync ) ; void FromRemoteFile( const Entry& remote, const DateTime& last_sync ) ; @@ -135,7 +135,7 @@ private : void DeleteRemote( http::Agent* http ) ; void AssignIDs( const Entry& remote ) ; - void SyncSelf( http::Agent* http, DateTime& sync_time ) ; + void SyncSelf( http::Agent* http ) ; private : std::string m_name ; From 3acd25cb553ad2a9950a47f98bc388fa21350b8e Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 23 Jul 2012 15:43:11 +0800 Subject: [PATCH 018/166] proper fix for (#98: missing reference count for Json::Get() ) --- libgrive/src/protocol/Json.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libgrive/src/protocol/Json.cc b/libgrive/src/protocol/Json.cc index 47ad94cd..5be3c06c 100644 --- a/libgrive/src/protocol/Json.cc +++ b/libgrive/src/protocol/Json.cc @@ -186,7 +186,7 @@ bool Json::Get( const std::string& key, Json& json ) const struct json_object *j = ::json_object_object_get( m_json, key.c_str() ) ; if ( j != 0 ) { - Json tmp( j, NotOwned() ) ; + Json tmp( j ) ; json.Swap( tmp ) ; return true ; } From 51e42914f3666ee6e0bc16a4c78f60b117265c24 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 23 Jul 2012 16:00:21 +0800 Subject: [PATCH 019/166] refreshing the token if HTTP 401 is returned (#83) --- README | 11 +++++++++-- libgrive/src/protocol/AuthAgent.cc | 12 ++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/README b/README index 5130b110..4b42df3e 100644 --- a/README +++ b/README @@ -1,4 +1,4 @@ -Grive 0.2.0 +Grive 0.3.0 9 June 2012 http://www.lbreda.com/grive/ @@ -14,7 +14,6 @@ There are a few things that Grive does not do at the moment: - wait for changes in file system to occur and upload. A sync is only performed when you run Grive. - symbolic links support - support for Google documents -- support for files >2GB Of course these will be added in the future, possibly the next release. @@ -48,3 +47,11 @@ current directory. It will also start downloading files from your Google Drive t your current directory. Enjoy! + +Version History: + +Grive v0.3: Bug fix release +Fixed bugs: + #93: missing reference count increment in one of the Json constructors + #82: retry for HTTP error 500 & 503 + #77: Fixed a bug where grive crashed on the first run. diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index b54f2324..15732a4c 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -125,6 +125,7 @@ std::string AuthAgent::Unescape( const std::string& str ) bool AuthAgent::CheckRetry( long response ) { + // HTTP 500 and 503 should be temperory. just wait a bit and retry if ( response == 500 || response == 503 ) { Log( "resquest failed due to temperory error: %1%. retrying in 5 seconds", @@ -133,6 +134,17 @@ bool AuthAgent::CheckRetry( long response ) os::Sleep( 5 ) ; return true ; } + + // HTTP 401 Unauthorized. the auth token has been expired. refresh it + else if ( response == 401 ) + { + Log( "resquest failed due to auth token expired: %1%. refreshing token", + response, log::warning ) ; + + m_auth.Refresh() ; + return true ; + } + else return false ; } From 2e9761e58e105bc53d231d59dbf2321a32e98e3a Mon Sep 17 00:00:00 2001 From: Stephen Cox Date: Mon, 23 Jul 2012 11:09:53 +0200 Subject: [PATCH 020/166] Removed duplicate "considered" in README --- README | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README b/README index 4b42df3e..fe6330e6 100644 --- a/README +++ b/README @@ -3,7 +3,7 @@ Grive 0.3.0 http://www.lbreda.com/grive/ -Grive can be considered still considered beta quality. It simply downloads all the files in your +Grive can be considered still beta quality. It simply downloads all the files in your Google Drive into the current directory. After you make some changes to the local files, run grive again and it will upload your changes back to your Google Drive. New files created locally or in Google Drive will be uploaded or downloaded respectively. Deleted files will also be "removed". From 5a0c6ae56e682022a2d8777ad51433322aaf20b0 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 26 Jul 2012 01:23:20 +0800 Subject: [PATCH 021/166] don't throw in CurlAgent for all HTTP errors (#82) --- libgrive/src/http/CurlAgent.cc | 3 +-- libgrive/src/http/Error.hh | 12 ++++----- libgrive/src/protocol/AuthAgent.cc | 42 +++++++++++++++++++++++++----- libgrive/src/protocol/AuthAgent.hh | 4 +++ 4 files changed, 46 insertions(+), 15 deletions(-) diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 85dfe38e..640e1eb5 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -156,12 +156,11 @@ long CurlAgent::ExecCurl( Trace( "HTTP response %1%", http_code ) ; ::curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, 0 ) ; - if ( curl_code != CURLE_OK || (http_code >= 400 && http_code < 500) ) + if ( curl_code != CURLE_OK ) { BOOST_THROW_EXCEPTION( Error() << CurlCode( curl_code ) - << HttpResponse( http_code ) << Url( url ) << expt::ErrMsg( error ) << HttpHeader( hdr ) diff --git a/libgrive/src/http/Error.hh b/libgrive/src/http/Error.hh index a513f5f3..9b41376e 100644 --- a/libgrive/src/http/Error.hh +++ b/libgrive/src/http/Error.hh @@ -29,16 +29,16 @@ struct Error : virtual Exception {} ; // CURL error code typedef boost::error_info CurlCode ; -// HTTP response code -typedef boost::error_info HttpResponse ; - -// HTTP response body -typedef boost::error_info HttpResponseText ; - // URL typedef boost::error_info Url ; // HTTP headers typedef boost::error_info HttpHeader ; +// HTTP response code +typedef boost::error_info HttpResponse ; + +// HTTP response body +typedef boost::error_info HttpResponseText ; + } } // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 15732a4c..9d0239b8 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -19,6 +19,7 @@ #include "AuthAgent.hh" +#include "http/Error.hh" #include "http/Header.hh" #include "util/log/Log.hh" #include "util/OS.hh" @@ -50,11 +51,13 @@ long AuthAgent::Put( Receivable *dest, const Header& hdr ) { + Header auth = AppendHeader(hdr) ; + long response ; while ( CheckRetry( - response = m_agent->Put(url, data, dest, AppendHeader(hdr)) ) ) ; + response = m_agent->Put(url, data, dest, auth) ) ) ; - return response ; + return CheckHttpResponse(response, url, auth) ; } long AuthAgent::Put( @@ -63,11 +66,13 @@ long AuthAgent::Put( Receivable *dest, const Header& hdr ) { + Header auth = AppendHeader(hdr) ; + long response ; while ( CheckRetry( response = m_agent->Put( url, file, dest, AppendHeader(hdr) ) ) ) ; - return response ; + return CheckHttpResponse(response, url, auth) ; } long AuthAgent::Get( @@ -75,11 +80,13 @@ long AuthAgent::Get( Receivable *dest, const Header& hdr ) { + Header auth = AppendHeader(hdr) ; + long response ; while ( CheckRetry( response = m_agent->Get( url, dest, AppendHeader(hdr) ) ) ) ; - return response ; + return CheckHttpResponse(response, url, auth) ; } long AuthAgent::Post( @@ -88,11 +95,13 @@ long AuthAgent::Post( Receivable *dest, const Header& hdr ) { + Header auth = AppendHeader(hdr) ; + long response ; while ( CheckRetry( response = m_agent->Post( url, data, dest, AppendHeader(hdr) ) ) ) ; - return response ; + return CheckHttpResponse(response, url, auth) ; } long AuthAgent::Custom( @@ -101,11 +110,13 @@ long AuthAgent::Custom( Receivable *dest, const Header& hdr ) { + Header auth = AppendHeader(hdr) ; + long response ; while ( CheckRetry( response = m_agent->Custom( method, url, dest, AppendHeader(hdr) ) ) ) ; - return response ; + return CheckHttpResponse(response, url, auth) ; } std::string AuthAgent::RedirLocation() const @@ -144,9 +155,26 @@ bool AuthAgent::CheckRetry( long response ) m_auth.Refresh() ; return true ; } - else return false ; } +long AuthAgent::CheckHttpResponse( + long response, + const std::string& url, + const http::Header& hdr ) +{ + // throw for other HTTP errors + if ( response >= 400 && response < 500 ) + { + BOOST_THROW_EXCEPTION( + Error() + << HttpResponse( response ) + << Url( url ) + << HttpHeader( hdr ) ) ; + } + + return response ; +} + } // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index 3b198c60..58900d18 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -73,6 +73,10 @@ public : private : http::Header AppendHeader( const http::Header& hdr ) const ; bool CheckRetry( long response ) ; + long CheckHttpResponse( + long response, + const std::string& url, + const http::Header& hdr ) ; private : OAuth2 m_auth ; From e375d92b82be53897e222ea93e3e124b15570c56 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 26 Jul 2012 01:29:47 +0800 Subject: [PATCH 022/166] version changes for v0.3 --- CMakeLists.txt | 2 +- README | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index ca39cb19..def7ca00 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! -set( GRIVE_VERSION "0.2.0" ) +set( GRIVE_VERSION "0.3.0-pre" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) diff --git a/README b/README index fe6330e6..84c3df2f 100644 --- a/README +++ b/README @@ -1,5 +1,5 @@ Grive 0.3.0 -9 June 2012 +25 July 2012 http://www.lbreda.com/grive/ From fd81cc1fe40f00cd6e65a1cc9930bad6b268712d Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Fri, 27 Jul 2012 00:17:44 +0800 Subject: [PATCH 023/166] added --log-xml for debugging (#80) --- grive/src/main.cc | 4 ++++ libgrive/src/drive/Drive.cc | 14 +++++++++----- libgrive/src/drive/Drive.hh | 5 ++--- 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index 47947f39..b017558c 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -82,6 +82,7 @@ int Main( int argc, char **argv ) ( "version,v", "Display Grive version" ) ( "auth,a", "Request authorization token" ) ( "verbose,V", "Verbose mode. Enable more messages than normal.") + ( "log-xml", "Log more HTTP responses as XML for debugging.") ( "debug,d", "Enable debug level messages. Implies -v.") ( "log,l", po::value(), "Set log output filename." ) ( "force,f", "Force grive to always download a file from Google Drive " @@ -146,6 +147,9 @@ int Main( int argc, char **argv ) { console_log->Enable( log::verbose ) ; } + + options.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; + if ( vm.count( "debug" ) ) { console_log->Enable( log::verbose ) ; diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index daf0a725..5151369d 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -27,7 +27,6 @@ #include "http/ResponseLog.hh" #include "http/XmlResponse.hh" #include "protocol/Json.hh" -// #include "protocol/OAuth2.hh" #include "util/Destroy.hh" #include "util/log/Log.hh" #include "xml/Node.hh" @@ -54,8 +53,9 @@ namespace } Drive::Drive( http::Agent *http, const Json& options ) : - m_http( http ), - m_state( state_file, options ) + m_http ( http ), + m_state ( state_file, options ), + m_log_xml ( options["log-xml"].Bool() ) { assert( m_http != 0 ) ; } @@ -137,7 +137,9 @@ void Drive::DetectChanges() Log( "Reading remote server file list", log::info ) ; Feed feed ; -// feed.EnableLog( "/tmp/file", ".xml" ) ; + if ( m_log_xml ) + feed.EnableLog( "/tmp/file", ".xml" ) ; + feed.Start( m_http, http::Header(), feed_base + "?showfolders=true&showroot=true" ) ; m_resume_link = feed.Root()["link"]. @@ -156,7 +158,9 @@ void Drive::DetectChanges() { Log( "Detecting changes from last sync", log::info ) ; Feed changes ; -// feed.EnableLog( "/tmp/changes", ".xml" ) ; + if ( m_log_xml ) + feed.EnableLog( "/tmp/changes", ".xml" ) ; + feed.Start( m_http, http::Header(), ChangesFeed(prev_stamp+1) ) ; std::for_each( diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 3641500c..11830b51 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -58,11 +58,10 @@ private : private : http::Agent *m_http ; -// OAuth2& m_auth ; -// http::Header m_http_hdr ; - std::string m_resume_link ; State m_state ; + + bool m_log_xml ; } ; } // end of namespace From d831b8d59f12a13c2a771b3b27532909164628e1 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Fri, 27 Jul 2012 00:45:53 +0800 Subject: [PATCH 024/166] support creating revisions with --new-rev (#87) --- grive/src/main.cc | 2 ++ libgrive/src/drive/Drive.cc | 11 +++++------ libgrive/src/drive/Drive.hh | 4 ++-- libgrive/src/drive/Resource.cc | 23 ++++++++++++----------- libgrive/src/drive/Resource.hh | 7 ++++--- libgrive/src/drive/State.cc | 5 +++-- libgrive/src/drive/State.hh | 2 +- libgrive/src/http/CurlAgent.cc | 9 ++++++--- 8 files changed, 35 insertions(+), 28 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index b017558c..cec9020f 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -83,6 +83,7 @@ int Main( int argc, char **argv ) ( "auth,a", "Request authorization token" ) ( "verbose,V", "Verbose mode. Enable more messages than normal.") ( "log-xml", "Log more HTTP responses as XML for debugging.") + ( "new-rev", "Create new revisions in server for updated files.") ( "debug,d", "Enable debug level messages. Implies -v.") ( "log,l", po::value(), "Set log output filename." ) ( "force,f", "Force grive to always download a file from Google Drive " @@ -149,6 +150,7 @@ int Main( int argc, char **argv ) } options.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; + options.Add( "new-rev", Json(vm.count("new-rev") > 0) ) ; if ( vm.count( "debug" ) ) { diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index 5151369d..5595fcce 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -26,7 +26,6 @@ #include "http/Agent.hh" #include "http/ResponseLog.hh" #include "http/XmlResponse.hh" -#include "protocol/Json.hh" #include "util/Destroy.hh" #include "util/log/Log.hh" #include "xml/Node.hh" @@ -55,7 +54,7 @@ namespace Drive::Drive( http::Agent *http, const Json& options ) : m_http ( http ), m_state ( state_file, options ), - m_log_xml ( options["log-xml"].Bool() ) + m_options ( options ) { assert( m_http != 0 ) ; } @@ -137,7 +136,7 @@ void Drive::DetectChanges() Log( "Reading remote server file list", log::info ) ; Feed feed ; - if ( m_log_xml ) + if ( m_options["log-xml"].Bool() ) feed.EnableLog( "/tmp/file", ".xml" ) ; feed.Start( m_http, http::Header(), feed_base + "?showfolders=true&showroot=true" ) ; @@ -158,7 +157,7 @@ void Drive::DetectChanges() { Log( "Detecting changes from last sync", log::info ) ; Feed changes ; - if ( m_log_xml ) + if ( m_options["log-xml"].Bool() ) feed.EnableLog( "/tmp/changes", ".xml" ) ; feed.Start( m_http, http::Header(), ChangesFeed(prev_stamp+1) ) ; @@ -172,7 +171,7 @@ void Drive::DetectChanges() void Drive::Update() { Log( "Synchronizing files", log::info ) ; - m_state.Sync( m_http ) ; + m_state.Sync( m_http, m_options ) ; UpdateChangeStamp( ) ; } @@ -180,7 +179,7 @@ void Drive::Update() void Drive::DryRun() { Log( "Synchronizing files (dry-run)", log::info ) ; - m_state.Sync( 0 ) ; + m_state.Sync( 0, m_options ) ; } void Drive::UpdateChangeStamp( ) diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 11830b51..3409953b 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -22,6 +22,7 @@ #include "State.hh" #include "http/Header.hh" +#include "protocol/Json.hh" #include "util/Exception.hh" #include @@ -35,7 +36,6 @@ namespace http } class Entry ; -class Json ; class Drive { @@ -61,7 +61,7 @@ private : std::string m_resume_link ; State m_state ; - bool m_log_xml ; + Json m_options ; } ; } // end of namespace diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index c044abc8..f63f99e3 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -365,23 +365,24 @@ Resource* Resource::FindChild( const std::string& name ) } // try to change the state to "sync" -void Resource::Sync( http::Agent *http, DateTime& sync_time ) +void Resource::Sync( http::Agent *http, DateTime& sync_time, const Json& options ) { assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced - SyncSelf( http ) ; - // we want the server sync time, so we will take the server time of the last file uploaded to store as the sync time - // m_mtime is updated to server modified time when the file is uploaded - sync_time = std::max(sync_time, m_mtime); + SyncSelf( http, options ) ; + + // we want the server sync time, so we will take the server time of the last file uploaded to store as the sync time + // m_mtime is updated to server modified time when the file is uploaded + sync_time = std::max(sync_time, m_mtime); // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) std::for_each( m_child.begin(), m_child.end(), - boost::bind( &Resource::Sync, _1, http, boost::ref(sync_time) ) ) ; + boost::bind( &Resource::Sync, _1, http, boost::ref(sync_time), options ) ) ; } -void Resource::SyncSelf( http::Agent* http ) +void Resource::SyncSelf( http::Agent* http, const Json& options ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync assert( IsRoot() || http == 0 || fs::is_directory( m_parent->Path() ) ) ; @@ -407,7 +408,7 @@ void Resource::SyncSelf( http::Agent* http ) case local_changed : Log( "sync %1% changed in local. uploading", path, log::info ) ; - if ( http != 0 && EditContent( http ) ) + if ( http != 0 && EditContent( http, options["new-rev"].Bool() ) ) m_state = sync ; break ; @@ -520,7 +521,7 @@ void Resource::Download( http::Agent* http, const fs::path& file ) const } } -bool Resource::EditContent( http::Agent* http ) +bool Resource::EditContent( http::Agent* http, bool new_rev ) { assert( http != 0 ) ; assert( m_parent != 0 ) ; @@ -533,10 +534,10 @@ bool Resource::EditContent( http::Agent* http ) return false ; } - return Upload( http, m_edit, false ) ; + return Upload( http, m_edit + (new_rev ? "?new-revision=true" : ""), false ) ; } -bool Resource::Create( http::Agent* http) +bool Resource::Create( http::Agent* http ) { assert( http != 0 ) ; assert( m_parent != 0 ) ; diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index b0cb2b8e..8b157f7b 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -35,6 +35,7 @@ namespace http } class Entry ; +class Json ; /*! \brief A resource can be a file or a folder in the google drive @@ -76,7 +77,7 @@ public : void FromRemote( const Entry& remote, const DateTime& last_sync ) ; void FromLocal( const DateTime& last_sync ) ; - void Sync( http::Agent* http, DateTime& sync_time ) ; + void Sync( http::Agent* http, DateTime& sync_time, const Json& options ) ; // children access iterator begin() const ; @@ -124,7 +125,7 @@ private : void SetState( State new_state ) ; void Download( http::Agent* http, const fs::path& file ) const ; - bool EditContent( http::Agent* http ) ; + bool EditContent( http::Agent* http, bool new_rev ) ; bool Create( http::Agent* http ) ; bool Upload( http::Agent* http, const std::string& link, bool post ) ; @@ -135,7 +136,7 @@ private : void DeleteRemote( http::Agent* http ) ; void AssignIDs( const Entry& remote ) ; - void SyncSelf( http::Agent* http ) ; + void SyncSelf( http::Agent* http, const Json& options ) ; private : std::string m_name ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index afab8420..8825f0c3 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -261,7 +261,7 @@ void State::Write( const fs::path& filename ) const fs << result ; } -void State::Sync( http::Agent *http ) +void State::Sync( http::Agent *http, const Json& options ) { // set the last sync time from the time returned by the server for the last file synced // if the sync time hasn't changed (i.e. now files have been uploaded) @@ -271,7 +271,8 @@ void State::Sync( http::Agent *http ) // TODO - WARNING - do we use the last sync time to compare to client file times // need to check if this introduces a new problem DateTime last_sync_time = m_last_sync; - m_res.Root()->Sync( http, last_sync_time ) ; + m_res.Root()->Sync( http, last_sync_time, options ) ; + if ( last_sync_time == m_last_sync ) { Trace( "nothing changed? %1%", m_last_sync ) ; diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index cd0590d4..9ddd435a 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -57,7 +57,7 @@ public : Resource* FindByID( const std::string& id ) ; Resource* Find( const fs::path& path ) ; - void Sync( http::Agent *http ) ; + void Sync( http::Agent *http, const Json& options ) ; iterator begin() ; iterator end() ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 640e1eb5..09bedee5 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -150,12 +150,15 @@ long CurlAgent::ExecCurl( dest->Clear() ; CURLcode curl_code = ::curl_easy_perform(curl); + // get the HTTTP response code long http_code = 0; - ::curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code); - + ::curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code); Trace( "HTTP response %1%", http_code ) ; - ::curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, 0 ) ; + // reset the curl buffer to prevent it from touch our "error" buffer + ::curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, 0 ) ; + + // only throw for libcurl errors if ( curl_code != CURLE_OK ) { BOOST_THROW_EXCEPTION( From e0383a843c4a28ce6ff8eada0fc8727dcaa53291 Mon Sep 17 00:00:00 2001 From: justin Date: Sun, 29 Jul 2012 17:53:02 +0100 Subject: [PATCH 025/166] remove unused code --- libgrive/src/drive/ResourceTree.cc | 23 ----------------------- libgrive/src/drive/ResourceTree.hh | 1 - libgrive/src/drive/State.cc | 5 ----- libgrive/src/drive/State.hh | 1 - 4 files changed, 30 deletions(-) diff --git a/libgrive/src/drive/ResourceTree.cc b/libgrive/src/drive/ResourceTree.cc index aedbd238..6562ba2c 100644 --- a/libgrive/src/drive/ResourceTree.cc +++ b/libgrive/src/drive/ResourceTree.cc @@ -111,29 +111,6 @@ const Resource* ResourceTree::FindByHref( const std::string& href ) const return i != map.end() ? *i : 0 ; } -/// Unlike other search functions, this one does not depend on the multi-index -/// container. It traverses the tree instead. -Resource* ResourceTree::FindByPath( const fs::path& path ) -{ - Resource *current = m_root ; - for ( fs::path::iterator i = path.begin() ; i != path.end() && current != 0 ; ++i ) - { - Trace( "path it = %1%", *i ) ; - - // current directory - if ( *i == "." ) - continue ; - - else if ( *i == ".." ) - current = current->Parent() ; - - else - current = current->FindChild( Path2Str(*i) ) ; - } - - return current ; -} - /// Reinsert should be called when the ID/HREF were updated bool ResourceTree::ReInsert( Resource *coll ) { diff --git a/libgrive/src/drive/ResourceTree.hh b/libgrive/src/drive/ResourceTree.hh index b7a2344e..c28482f6 100644 --- a/libgrive/src/drive/ResourceTree.hh +++ b/libgrive/src/drive/ResourceTree.hh @@ -74,7 +74,6 @@ public : Resource* FindByHref( const std::string& href ) ; const Resource* FindByHref( const std::string& href ) const ; - Resource* FindByPath( const fs::path& path ) ; Resource* FindByID( const std::string& id ) ; bool ReInsert( Resource *coll ) ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 8825f0c3..3a6e2759 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -213,11 +213,6 @@ Resource* State::FindByHref( const std::string& href ) return m_res.FindByHref( href ) ; } -Resource* State::Find( const fs::path& path ) -{ - return m_res.FindByPath( path ) ; -} - State::iterator State::begin() { return m_res.begin() ; diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index 9ddd435a..a0e12132 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -55,7 +55,6 @@ public : Resource* FindByHref( const std::string& href ) ; Resource* FindByID( const std::string& id ) ; - Resource* Find( const fs::path& path ) ; void Sync( http::Agent *http, const Json& options ) ; From 034891a6103282dfe175eba10e882303c2d211d5 Mon Sep 17 00:00:00 2001 From: justin Date: Sun, 29 Jul 2012 18:17:10 +0100 Subject: [PATCH 026/166] issue 86 allow sync folder to be specified, move Config class and add basic unit test --- grive/src/main.cc | 93 ++++++++++++++-------- libgrive/src/drive/Drive.cc | 9 ++- libgrive/src/drive/Drive.hh | 4 +- libgrive/src/drive/Resource.cc | 4 +- libgrive/src/drive/Resource.hh | 2 +- libgrive/src/drive/ResourceTree.cc | 4 +- libgrive/src/drive/ResourceTree.hh | 2 +- libgrive/src/drive/State.cc | 5 +- libgrive/src/drive/State.hh | 2 +- {grive/src => libgrive/src/util}/Config.cc | 25 +++--- {grive/src => libgrive/src/util}/Config.hh | 11 +-- libgrive/test/UnitTest.cc | 2 + libgrive/test/drive/ResourceTest.cc | 7 +- libgrive/test/util/ConfigTest.cc | 57 +++++++++++++ libgrive/test/util/ConfigTest.hh | 47 +++++++++++ 15 files changed, 206 insertions(+), 68 deletions(-) rename {grive/src => libgrive/src/util}/Config.cc (69%) rename {grive/src => libgrive/src/util}/Config.hh (85%) create mode 100644 libgrive/test/util/ConfigTest.cc create mode 100644 libgrive/test/util/ConfigTest.hh diff --git a/grive/src/main.cc b/grive/src/main.cc index cec9020f..d576001e 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -17,7 +17,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "Config.hh" +#include "util/Config.hh" #include "drive/Drive.hh" @@ -46,6 +46,9 @@ const std::string client_id = "22314510474.apps.googleusercontent.com" ; const std::string client_secret = "bl4ufi89h-9MkFlypcI7R785" ; +const std::string defaultRootFolder = "."; +const std::string defaultConfigFileName = ".grive"; +const char *configFileEnvironmentVariable = "GR_CONFIG"; using namespace gr ; @@ -66,8 +69,8 @@ int Main( int argc, char **argv ) { InitGCrypt() ; - Config config ; - + std::string rootFolder = defaultRootFolder; + std::auto_ptr comp_log(new log::CompositeLog) ; LogBase* console_log = comp_log->Add( std::auto_ptr( new log::DefaultLog ) ) ; @@ -81,6 +84,7 @@ int Main( int argc, char **argv ) ( "help,h", "Produce help message" ) ( "version,v", "Display Grive version" ) ( "auth,a", "Request authorization token" ) + ( "path,p", po::value(), "Path to sync") ( "verbose,V", "Verbose mode. Enable more messages than normal.") ( "log-xml", "Log more HTTP responses as XML for debugging.") ( "new-rev", "Create new revisions in server for updated files.") @@ -101,27 +105,9 @@ int Main( int argc, char **argv ) std::cout << desc << std::endl ; return 0 ; } - if ( vm.count( "auth" ) ) - { - std::cout - << "-----------------------\n" - << "Please go to this URL and get an authentication code:\n\n" - << OAuth2::MakeAuthURL( client_id ) - << std::endl ; - - std::cout - << "\n-----------------------\n" - << "Please input the authentication code here: " << std::endl ; - std::string code ; - std::cin >> code ; - - OAuth2 token( client_id, client_secret ) ; - token.Auth( code ) ; - - // save to config - config.Get().Add( "refresh_token", Json( token.RefreshToken() ) ) ; - config.Save() ; - } + + boost::shared_ptr config ; + if ( vm.count( "log" ) ) { std::auto_ptr file_log(new log::DefaultLog( vm["log"].as() )) ; @@ -138,12 +124,6 @@ int Main( int argc, char **argv ) comp_log->Add( file_log ) ; } - if ( vm.count( "version" ) ) - { - std::cout - << "grive version " << VERSION << ' ' << __DATE__ << ' ' << __TIME__ << std::endl ; - return 0 ; - } if ( vm.count( "verbose" ) ) { console_log->Enable( log::verbose ) ; @@ -157,6 +137,52 @@ int Main( int argc, char **argv ) console_log->Enable( log::verbose ) ; console_log->Enable( log::debug ) ; } + + // config file will be (in order of preference) + // value specified in environment string + // value specified in defaultConfigFileName in path from commandline --path + // value specified in defaultConfigFileName in current directory + const char *envConfigFileName = ::getenv( configFileEnvironmentVariable ) ; + if (envConfigFileName) { + config.reset(new Config(envConfigFileName)); + + } else if ( vm.count( "path" ) ) { + rootFolder = vm["path"].as(); + config.reset(new Config( fs::path(rootFolder) / fs::path(defaultConfigFileName) )); + + } else { + config.reset(new Config( defaultConfigFileName) ); + } + + Log( "config file name %1%", config->ConfigFile().string(), log::verbose ); + + if ( vm.count( "auth" ) ) + { + std::cout + << "-----------------------\n" + << "Please go to this URL and get an authentication code:\n\n" + << OAuth2::MakeAuthURL( client_id ) + << std::endl ; + + std::cout + << "\n-----------------------\n" + << "Please input the authentication code here: " << std::endl ; + std::string code ; + std::cin >> code ; + + OAuth2 token( client_id, client_secret ) ; + token.Auth( code ) ; + + // save to config + config->Get().Add( "refresh_token", Json( token.RefreshToken() ) ) ; + config->Save() ; + } + if ( vm.count( "version" ) ) + { + std::cout + << "grive version " << VERSION << ' ' << __DATE__ << ' ' << __TIME__ << std::endl ; + return 0 ; + } if ( vm.count( "force" ) ) { options.Add( "force", Json(true) ) ; @@ -167,7 +193,7 @@ int Main( int argc, char **argv ) std::string refresh_token ; try { - refresh_token = config.Get()["refresh_token"].Str() ; + refresh_token = config->Get()["refresh_token"].Str() ; } catch ( Exception& e ) { @@ -181,7 +207,8 @@ int Main( int argc, char **argv ) OAuth2 token( refresh_token, client_id, client_secret ) ; AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; - Drive drive( &agent, options ) ; + + Drive drive( &agent, options, rootFolder ) ; drive.DetectChanges() ; if ( vm.count( "dry-run" ) == 0 ) @@ -192,7 +219,7 @@ int Main( int argc, char **argv ) else drive.DryRun() ; - config.Save() ; + config->Save() ; Log( "Finished!", log::info ) ; return 0 ; } diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index 5595fcce..028e62c7 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -51,9 +51,10 @@ namespace const std::string state_file = ".grive_state" ; } -Drive::Drive( http::Agent *http, const Json& options ) : - m_http ( http ), - m_state ( state_file, options ), +Drive::Drive( http::Agent *http, const Json& options, const std::string &rootFolder) : + m_http( http ), + m_state( fs::path(rootFolder), fs::path(rootFolder) / fs::path(state_file), options ), + m_rootFolder(rootFolder), m_options ( options ) { assert( m_http != 0 ) ; @@ -127,7 +128,7 @@ void Drive::SyncFolders( ) void Drive::DetectChanges() { Log( "Reading local directories", log::info ) ; - m_state.FromLocal( "." ) ; + m_state.FromLocal( m_rootFolder ) ; long prev_stamp = m_state.ChangeStamp() ; Trace( "previous change stamp is %1%", prev_stamp ) ; diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 3409953b..bcae37c0 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -40,7 +40,7 @@ class Entry ; class Drive { public : - Drive( http::Agent *http, const Json& options ) ; + Drive( http::Agent *http, const Json& options, const std::string &rootFolder ) ; void DetectChanges() ; void Update() ; @@ -60,7 +60,7 @@ private : http::Agent *m_http ; std::string m_resume_link ; State m_state ; - + fs::path m_rootFolder; Json m_options ; } ; diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index f63f99e3..53e4fc4e 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -58,8 +58,8 @@ const std::string xml_meta = /// default constructor creates the root folder -Resource::Resource() : - m_name ( "." ), +Resource::Resource(const fs::path& rootFolder) : + m_name ( rootFolder.string() ), m_kind ( "folder" ), m_id ( "folder:root" ), m_href ( root_href ), diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 8b157f7b..9406c6f7 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -51,7 +51,7 @@ public : typedef Children::const_iterator iterator ; public : - Resource() ; + Resource(const fs::path& rootFolder) ; Resource( const std::string& name, const std::string& kind ) ; // default copy ctor & op= are fine diff --git a/libgrive/src/drive/ResourceTree.cc b/libgrive/src/drive/ResourceTree.cc index 6562ba2c..57615792 100644 --- a/libgrive/src/drive/ResourceTree.cc +++ b/libgrive/src/drive/ResourceTree.cc @@ -31,8 +31,8 @@ namespace gr { using namespace details ; -ResourceTree::ResourceTree( ) : - m_root( new Resource ) +ResourceTree::ResourceTree( const fs::path& rootFolder ) : + m_root(new Resource(rootFolder)) { m_set.insert( m_root ) ; } diff --git a/libgrive/src/drive/ResourceTree.hh b/libgrive/src/drive/ResourceTree.hh index c28482f6..1edbc8c3 100644 --- a/libgrive/src/drive/ResourceTree.hh +++ b/libgrive/src/drive/ResourceTree.hh @@ -64,7 +64,7 @@ public : typedef details::Set::iterator iterator ; public : - ResourceTree( ) ; + ResourceTree( const fs::path& rootFolder ) ; ResourceTree( const ResourceTree& fs ) ; ~ResourceTree( ) ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 3a6e2759..b69b756b 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -32,8 +32,9 @@ namespace gr { -State::State( const fs::path& filename, const Json& options ) : - m_cstamp( -1 ) +State::State( const fs::path& rootFolder, const fs::path& filename, const Json& options ) : + m_cstamp( -1 ), + m_res(rootFolder) { Read( filename ) ; diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index a0e12132..2cc2405d 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -43,7 +43,7 @@ public : typedef ResourceTree::iterator iterator ; public : - explicit State( const fs::path& filename, const Json& options ) ; + explicit State( const fs::path& rootFolder, const fs::path& filename, const Json& options ) ; ~State() ; void FromLocal( const fs::path& p ) ; diff --git a/grive/src/Config.cc b/libgrive/src/util/Config.cc similarity index 69% rename from grive/src/Config.cc rename to libgrive/src/util/Config.cc index 6a66d569..d74307b5 100644 --- a/grive/src/Config.cc +++ b/libgrive/src/util/Config.cc @@ -21,26 +21,28 @@ #include "util/StdioFile.hh" +#include #include -namespace gr { +using namespace gr; -const std::string& Config::Filename() +Config::Config(const fs::path& configFile) + : m_configFile(configFile) + , m_cfg( Read() ) { - static const char *env_cfg = ::getenv( "GR_CONFIG" ) ; - static const std::string filename = (env_cfg != 0) ? env_cfg : ".grive" ; - - return filename ; + if (configFile.empty()) { + throw Error() << expt::ErrMsg("Config cannot be initalised with an empty string."); + } } -Config::Config() : - m_cfg( Read( Filename() ) ) +const fs::path& Config::ConfigFile() const { + return m_configFile ; } void Config::Save( ) { - StdioFile file( Filename(), 0600 ) ; + StdioFile file( m_configFile.string(), 0600 ) ; m_cfg.Write( file ) ; } @@ -49,11 +51,11 @@ Json& Config::Get() return m_cfg ; } -Json Config::Read( const std::string& filename ) +Json Config::Read() { try { - return Json::ParseFile( filename ) ; + return Json::ParseFile( m_configFile.string() ) ; } catch ( Exception& e ) { @@ -61,4 +63,3 @@ Json Config::Read( const std::string& filename ) } } -} // end of namespace diff --git a/grive/src/Config.hh b/libgrive/src/util/Config.hh similarity index 85% rename from grive/src/Config.hh rename to libgrive/src/util/Config.hh index 11188fa9..63af600e 100644 --- a/grive/src/Config.hh +++ b/libgrive/src/util/Config.hh @@ -19,7 +19,8 @@ #pragma once -#include "util/Exception.hh" +#include "Exception.hh" +#include "FileSystem.hh" #include "protocol/Json.hh" namespace gr { @@ -30,17 +31,17 @@ public : struct Error : virtual Exception {} ; typedef boost::error_info File ; - static const std::string& Filename() ; - - Config() ; + Config(const fs::path& configFile) ; Json& Get() ; void Save() ; + const fs::path &ConfigFile() const; private : - Json Read( const std::string& filename ) ; + Json Read() ; private : + const fs::path m_configFile; Json m_cfg ; } ; diff --git a/libgrive/test/UnitTest.cc b/libgrive/test/UnitTest.cc index f599edf5..19dcf503 100644 --- a/libgrive/test/UnitTest.cc +++ b/libgrive/test/UnitTest.cc @@ -27,6 +27,7 @@ #include "drive/StateTest.hh" #include "util/DateTimeTest.hh" #include "util/FunctionTest.hh" +#include "util/ConfigTest.hh" #include "util/SignalHandlerTest.hh" #include "xml/NodeTest.hh" @@ -43,6 +44,7 @@ int main( int argc, char **argv ) runner.addTest( ResourceTreeTest::suite( ) ) ; runner.addTest( DateTimeTest::suite( ) ) ; runner.addTest( FunctionTest::suite( ) ) ; + runner.addTest( ConfigTest::suite( ) ) ; runner.addTest( SignalHandlerTest::suite( ) ) ; runner.addTest( NodeTest::suite( ) ) ; runner.run(); diff --git a/libgrive/test/drive/ResourceTest.cc b/libgrive/test/drive/ResourceTest.cc index 1199561f..e8190542 100644 --- a/libgrive/test/drive/ResourceTest.cc +++ b/libgrive/test/drive/ResourceTest.cc @@ -38,9 +38,10 @@ ResourceTest::ResourceTest( ) void ResourceTest::TestRootPath() { - Resource root ; + std::string rootFolder = "/home/usr/grive/grive"; + Resource root(rootFolder) ; CPPUNIT_ASSERT( root.IsRoot() ) ; - GRUT_ASSERT_EQUAL( root.Path(), fs::path( "." ) ) ; + GRUT_ASSERT_EQUAL( root.Path(), fs::path( rootFolder ) ) ; } void ResourceTest::TestNormal( ) @@ -60,7 +61,7 @@ void ResourceTest::TestNormal( ) Entry remote( entry ) ; subject.FromRemote( remote, DateTime() ) ; - GRUT_ASSERT_EQUAL( subject.StateStr(), "local_changed" ) ; + GRUT_ASSERT_EQUAL( "local_changed", subject.StateStr() ) ; } diff --git a/libgrive/test/util/ConfigTest.cc b/libgrive/test/util/ConfigTest.cc new file mode 100644 index 00000000..33a427d9 --- /dev/null +++ b/libgrive/test/util/ConfigTest.cc @@ -0,0 +1,57 @@ +/* +grive: an GPL program to sync a local directory with Google Drive +Copyright (C) 2012 Wan Wai Ho + +This program is free software; you can redistribute it and/or +modify it under the terms of the GNU General Public License +as published by the Free Software Foundation version 2 +of the License. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program; if not, write to the Free Software +Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "ConfigTest.hh" + +#include "Assert.hh" + +#include "util/Config.hh" +#include "protocol/Json.hh" +#include "util/log/Log.hh" + +#include + +using namespace grut; +using namespace gr ; + +ConfigTest::ConfigTest( ) +{ +} + +void ConfigTest::TestInitialiseWithEmptyString( ) +{ + Config config(""); + GRUT_ASSERT_EQUAL( "/home/.grive", config.ConfigFile().string()) ; +} + +void ConfigTest::TestInitialiseWithString( ) +{ + Config config("/home/.grive"); + GRUT_ASSERT_EQUAL( "/home/.grive", config.ConfigFile().string()) ; +} + +void ConfigTest::TestInitialiseWithFileSystemPath( ) +{ + fs::path path("/home"); + fs::path file(".grive"); + Config config(path / file); + GRUT_ASSERT_EQUAL( "/home/.grive", config.ConfigFile().string()); +} + + diff --git a/libgrive/test/util/ConfigTest.hh b/libgrive/test/util/ConfigTest.hh new file mode 100644 index 00000000..90325743 --- /dev/null +++ b/libgrive/test/util/ConfigTest.hh @@ -0,0 +1,47 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "util/Config.hh" +#include +#include + +namespace grut { + +class ConfigTest : public CppUnit::TestFixture +{ +public : + ConfigTest( ) ; + + // declare suit function + CPPUNIT_TEST_SUITE( ConfigTest ) ; + CPPUNIT_TEST_EXCEPTION( TestInitialiseWithEmptyString, gr::Config::Error ) ; + CPPUNIT_TEST( TestInitialiseWithString ) ; + CPPUNIT_TEST( TestInitialiseWithFileSystemPath ) ; + CPPUNIT_TEST_SUITE_END(); + +private : + void TestInitialiseWithEmptyString( ); + void TestInitialiseWithString( ); + void TestInitialiseWithFileSystemPath( ); +} ; + +} // end of namespace + From 0b2a8c0a546bd562bf4037757b652356a4e0a7f8 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 30 Jul 2012 15:24:47 +0800 Subject: [PATCH 027/166] clean up on #109 --- README | 7 +- grive/src/main.cc | 124 ++++++++++++++++++------------------ libgrive/CMakeLists.txt | 3 +- libgrive/src/util/Config.cc | 18 ++++-- libgrive/src/util/Config.hh | 6 +- 5 files changed, 85 insertions(+), 73 deletions(-) diff --git a/README b/README index 84c3df2f..a713a75c 100644 --- a/README +++ b/README @@ -50,8 +50,13 @@ Enjoy! Version History: -Grive v0.3: Bug fix release +Grive v0.3: Bug fix & minor feature release Fixed bugs: #93: missing reference count increment in one of the Json constructors #82: retry for HTTP error 500 & 503 #77: Fixed a bug where grive crashed on the first run. + +New features: + #87: support for revisions + #86: partial sync (contributed by justin at tierramedia.com) + \ No newline at end of file diff --git a/grive/src/main.cc b/grive/src/main.cc index d576001e..9516a2cb 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -51,6 +51,7 @@ const std::string defaultConfigFileName = ".grive"; const char *configFileEnvironmentVariable = "GR_CONFIG"; using namespace gr ; +namespace po = boost::program_options; // libgcrypt insist this to be done in application, not library void InitGCrypt() @@ -65,18 +66,46 @@ void InitGCrypt() gcry_control(GCRYCTL_INITIALIZATION_FINISHED, 0); } -int Main( int argc, char **argv ) +void InitLog( const po::variables_map& vm ) { - InitGCrypt() ; - - std::string rootFolder = defaultRootFolder; - std::auto_ptr comp_log(new log::CompositeLog) ; LogBase* console_log = comp_log->Add( std::auto_ptr( new log::DefaultLog ) ) ; + + if ( vm.count( "log" ) ) + { + std::auto_ptr file_log(new log::DefaultLog( vm["log"].as() )) ; + file_log->Enable( log::debug ) ; + file_log->Enable( log::verbose ) ; + file_log->Enable( log::info ) ; + file_log->Enable( log::warning ) ; + file_log->Enable( log::error ) ; + file_log->Enable( log::critical ) ; + + // log grive version to log file + file_log->Log( log::Fmt("grive version " VERSION " " __DATE__ " " __TIME__), log::verbose ) ; + file_log->Log( log::Fmt("current time: %1%") % DateTime::Now(), log::verbose ) ; + + comp_log->Add( file_log ) ; + } - Json options ; + if ( vm.count( "verbose" ) ) + { + console_log->Enable( log::verbose ) ; + } - namespace po = boost::program_options; + if ( vm.count( "debug" ) ) + { + console_log->Enable( log::verbose ) ; + console_log->Enable( log::debug ) ; + } + LogBase::Inst( std::auto_ptr(comp_log.release()) ) ; +} + +int Main( int argc, char **argv ) +{ + InitGCrypt() ; + + std::string rootFolder = defaultRootFolder; // construct the program options po::options_description desc( "Grive options" ); @@ -105,56 +134,41 @@ int Main( int argc, char **argv ) std::cout << desc << std::endl ; return 0 ; } - - boost::shared_ptr config ; - - if ( vm.count( "log" ) ) - { - std::auto_ptr file_log(new log::DefaultLog( vm["log"].as() )) ; - file_log->Enable( log::debug ) ; - file_log->Enable( log::verbose ) ; - file_log->Enable( log::info ) ; - file_log->Enable( log::warning ) ; - file_log->Enable( log::error ) ; - file_log->Enable( log::critical ) ; - - // log grive version to log file - file_log->Log( log::Fmt("grive version " VERSION " " __DATE__ " " __TIME__), log::verbose ) ; - file_log->Log( log::Fmt("current time: %1%") % DateTime::Now(), log::verbose ) ; - - comp_log->Add( file_log ) ; - } - if ( vm.count( "verbose" ) ) + else if ( vm.count( "version" ) ) { - console_log->Enable( log::verbose ) ; + std::cout + << "grive version " << VERSION << ' ' << __DATE__ << ' ' << __TIME__ << std::endl ; + return 0 ; } + + // initialize logging + InitLog(vm) ; - options.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; - options.Add( "new-rev", Json(vm.count("new-rev") > 0) ) ; + boost::shared_ptr config ; - if ( vm.count( "debug" ) ) - { - console_log->Enable( log::verbose ) ; - console_log->Enable( log::debug ) ; - } - - // config file will be (in order of preference) - // value specified in environment string - // value specified in defaultConfigFileName in path from commandline --path - // value specified in defaultConfigFileName in current directory + // config file will be (in order of preference) + // value specified in environment string + // value specified in defaultConfigFileName in path from commandline --path + // value specified in defaultConfigFileName in current directory const char *envConfigFileName = ::getenv( configFileEnvironmentVariable ) ; - if (envConfigFileName) { - config.reset(new Config(envConfigFileName)); + if (envConfigFileName) { + config.reset(new Config(envConfigFileName)); - } else if ( vm.count( "path" ) ) { - rootFolder = vm["path"].as(); - config.reset(new Config( fs::path(rootFolder) / fs::path(defaultConfigFileName) )); + } else if ( vm.count( "path" ) ) { + rootFolder = vm["path"].as(); + config.reset(new Config( fs::path(rootFolder) / fs::path(defaultConfigFileName) )); - } else { - config.reset(new Config( defaultConfigFileName) ); - } + } else { + config.reset(new Config( defaultConfigFileName) ); + } - Log( "config file name %1%", config->ConfigFile().string(), log::verbose ); + // misc options + Json options ; + options.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; + options.Add( "new-rev", Json(vm.count("new-rev") > 0) ) ; + options.Add( "force", Json(vm.count("force") > 0 ) ) ; + + Log( "config file name %1%", config->ConfigFile().string(), log::verbose ); if ( vm.count( "auth" ) ) { @@ -177,18 +191,6 @@ int Main( int argc, char **argv ) config->Get().Add( "refresh_token", Json( token.RefreshToken() ) ) ; config->Save() ; } - if ( vm.count( "version" ) ) - { - std::cout - << "grive version " << VERSION << ' ' << __DATE__ << ' ' << __TIME__ << std::endl ; - return 0 ; - } - if ( vm.count( "force" ) ) - { - options.Add( "force", Json(true) ) ; - } - - LogBase::Inst( std::auto_ptr(comp_log.release()) ) ; std::string refresh_token ; try diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 5008a3ca..a9d6018a 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -6,7 +6,7 @@ find_package(LibGcrypt REQUIRED) find_package(JSONC REQUIRED) find_package(CURL REQUIRED) find_package(EXPAT REQUIRED) -find_package(Boost 1.40.0 COMPONENTS filesystem system REQUIRED) +find_package(Boost 1.40.0 COMPONENTS program_options filesystem system REQUIRED) find_package(BFD) find_package(CppUnit) find_package(Iberty) @@ -126,6 +126,7 @@ IF ( CPPUNIT_FOUND ) target_link_libraries( unittest grive ${CPPUNIT_LIBRARY} + ${Boost_LIBRARIES} ) ENDIF ( CPPUNIT_FOUND ) diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index d74307b5..62a8333a 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -21,18 +21,21 @@ #include "util/StdioFile.hh" +#include + #include #include -using namespace gr; +namespace gr { -Config::Config(const fs::path& configFile) - : m_configFile(configFile) - , m_cfg( Read() ) +Config::Config(const fs::path& configFile) : + m_configFile(configFile), + m_cfg( Read() ) { - if (configFile.empty()) { - throw Error() << expt::ErrMsg("Config cannot be initalised with an empty string."); - } + if (configFile.empty()) + { + throw Error() << expt::ErrMsg("Config cannot be initalised with an empty string."); + } } const fs::path& Config::ConfigFile() const @@ -63,3 +66,4 @@ Json Config::Read() } } +} // end of namespace diff --git a/libgrive/src/util/Config.hh b/libgrive/src/util/Config.hh index 63af600e..30ef116b 100644 --- a/libgrive/src/util/Config.hh +++ b/libgrive/src/util/Config.hh @@ -35,14 +35,14 @@ public : Json& Get() ; void Save() ; - const fs::path &ConfigFile() const; + const fs::path &ConfigFile() const; private : Json Read() ; private : - const fs::path m_configFile; - Json m_cfg ; + const fs::path m_configFile; + Json m_cfg ; } ; } // end of namespace From bec83626395d45286fd5673446ce7a27b93f636f Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 30 Jul 2012 16:43:36 +0800 Subject: [PATCH 028/166] added template contructor for object --- libgrive/src/protocol/Json.cc | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/libgrive/src/protocol/Json.cc b/libgrive/src/protocol/Json.cc index 5be3c06c..7012db77 100644 --- a/libgrive/src/protocol/Json.cc +++ b/libgrive/src/protocol/Json.cc @@ -101,6 +101,17 @@ Json::Json( const bool& b ) : BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json bool" ) ) ; } +template <> +Json::Json( const Object& obj ) : + m_json( ::json_object_new_object() ) +{ + if ( m_json == 0 ) + BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json object" ) ) ; + + for ( Object::const_iterator i = obj.begin() ; i != obj.end() ; ++i ) + Add( i->first, i->second ) ; +} + Json::Json( struct json_object *json, NotOwned ) : m_json( json ) { @@ -198,7 +209,7 @@ void Json::Add( const std::string& key, const Json& json ) { assert( m_json != 0 ) ; assert( json.m_json != 0 ) ; - + ::json_object_get( json.m_json ) ; ::json_object_object_add( m_json, key.c_str(), json.m_json ) ; } From 612b35a0906fd2a05dd6902631a314e1e914de3d Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 30 Jul 2012 16:43:51 +0800 Subject: [PATCH 029/166] fixed compiler warnings --- libgrive/src/drive/State.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index b69b756b..cd2aec6e 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -33,8 +33,8 @@ namespace gr { State::State( const fs::path& rootFolder, const fs::path& filename, const Json& options ) : - m_cstamp( -1 ), - m_res(rootFolder) + m_res(rootFolder), + m_cstamp( -1 ) { Read( filename ) ; From 804fe923f2a64862c3588fa216013484cde8e2f7 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 30 Jul 2012 16:52:59 +0800 Subject: [PATCH 030/166] refactored the config file part from #109 --- grive/src/main.cc | 42 ++++--------------- libgrive/src/drive/Drive.cc | 10 ++--- libgrive/src/drive/Drive.hh | 4 +- libgrive/src/drive/Resource.cc | 4 +- libgrive/src/drive/Resource.hh | 2 +- libgrive/src/drive/State.cc | 6 +-- libgrive/src/drive/State.hh | 2 +- libgrive/src/util/Config.cc | 69 +++++++++++++++++++++++++------- libgrive/src/util/Config.hh | 32 ++++++++++++--- libgrive/test/util/ConfigTest.cc | 16 ++++---- 10 files changed, 111 insertions(+), 76 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index 9516a2cb..cf3a3d1a 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -46,9 +46,6 @@ const std::string client_id = "22314510474.apps.googleusercontent.com" ; const std::string client_secret = "bl4ufi89h-9MkFlypcI7R785" ; -const std::string defaultRootFolder = "."; -const std::string defaultConfigFileName = ".grive"; -const char *configFileEnvironmentVariable = "GR_CONFIG"; using namespace gr ; namespace po = boost::program_options; @@ -105,8 +102,6 @@ int Main( int argc, char **argv ) { InitGCrypt() ; - std::string rootFolder = defaultRootFolder; - // construct the program options po::options_description desc( "Grive options" ); desc.add_options() @@ -129,6 +124,7 @@ int Main( int argc, char **argv ) po::store(po::parse_command_line( argc, argv, desc), vm ); po::notify(vm); + // simple commands that doesn't require log or config if ( vm.count("help") ) { std::cout << desc << std::endl ; @@ -144,31 +140,9 @@ int Main( int argc, char **argv ) // initialize logging InitLog(vm) ; - boost::shared_ptr config ; - - // config file will be (in order of preference) - // value specified in environment string - // value specified in defaultConfigFileName in path from commandline --path - // value specified in defaultConfigFileName in current directory - const char *envConfigFileName = ::getenv( configFileEnvironmentVariable ) ; - if (envConfigFileName) { - config.reset(new Config(envConfigFileName)); - - } else if ( vm.count( "path" ) ) { - rootFolder = vm["path"].as(); - config.reset(new Config( fs::path(rootFolder) / fs::path(defaultConfigFileName) )); - - } else { - config.reset(new Config( defaultConfigFileName) ); - } - - // misc options - Json options ; - options.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; - options.Add( "new-rev", Json(vm.count("new-rev") > 0) ) ; - options.Add( "force", Json(vm.count("force") > 0 ) ) ; + Config config(vm) ; - Log( "config file name %1%", config->ConfigFile().string(), log::verbose ); + Log( "config file name %1%", config.Filename(), log::verbose ); if ( vm.count( "auth" ) ) { @@ -188,14 +162,14 @@ int Main( int argc, char **argv ) token.Auth( code ) ; // save to config - config->Get().Add( "refresh_token", Json( token.RefreshToken() ) ) ; - config->Save() ; + config.Set( "refresh_token", Json( token.RefreshToken() ) ) ; + config.Save() ; } std::string refresh_token ; try { - refresh_token = config->Get()["refresh_token"].Str() ; + refresh_token = config.Get("refresh_token").Str() ; } catch ( Exception& e ) { @@ -210,7 +184,7 @@ int Main( int argc, char **argv ) OAuth2 token( refresh_token, client_id, client_secret ) ; AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; - Drive drive( &agent, options, rootFolder ) ; + Drive drive( &agent, config.GetAll() ) ; drive.DetectChanges() ; if ( vm.count( "dry-run" ) == 0 ) @@ -221,7 +195,7 @@ int Main( int argc, char **argv ) else drive.DryRun() ; - config->Save() ; + config.Save() ; Log( "Finished!", log::info ) ; return 0 ; } diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index 028e62c7..82cfc749 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -51,10 +51,10 @@ namespace const std::string state_file = ".grive_state" ; } -Drive::Drive( http::Agent *http, const Json& options, const std::string &rootFolder) : - m_http( http ), - m_state( fs::path(rootFolder), fs::path(rootFolder) / fs::path(state_file), options ), - m_rootFolder(rootFolder), +Drive::Drive( http::Agent *http, const Json& options ) : + m_http ( http ), + m_root ( options["path"].Str() ), + m_state ( m_root / state_file, options ), m_options ( options ) { assert( m_http != 0 ) ; @@ -128,7 +128,7 @@ void Drive::SyncFolders( ) void Drive::DetectChanges() { Log( "Reading local directories", log::info ) ; - m_state.FromLocal( m_rootFolder ) ; + m_state.FromLocal( m_root ) ; long prev_stamp = m_state.ChangeStamp() ; Trace( "previous change stamp is %1%", prev_stamp ) ; diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index bcae37c0..5424f44f 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -40,7 +40,7 @@ class Entry ; class Drive { public : - Drive( http::Agent *http, const Json& options, const std::string &rootFolder ) ; + Drive( http::Agent *http, const Json& options ) ; void DetectChanges() ; void Update() ; @@ -59,8 +59,8 @@ private : private : http::Agent *m_http ; std::string m_resume_link ; + fs::path m_root ; State m_state ; - fs::path m_rootFolder; Json m_options ; } ; diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 53e4fc4e..e64bd0fe 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -58,8 +58,8 @@ const std::string xml_meta = /// default constructor creates the root folder -Resource::Resource(const fs::path& rootFolder) : - m_name ( rootFolder.string() ), +Resource::Resource(const fs::path& root_folder) : + m_name ( root_folder.string() ), m_kind ( "folder" ), m_id ( "folder:root" ), m_href ( root_href ), diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 9406c6f7..5d4f0348 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -51,7 +51,7 @@ public : typedef Children::const_iterator iterator ; public : - Resource(const fs::path& rootFolder) ; + Resource(const fs::path& root_folder) ; Resource( const std::string& name, const std::string& kind ) ; // default copy ctor & op= are fine diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index cd2aec6e..43e63679 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -32,9 +32,9 @@ namespace gr { -State::State( const fs::path& rootFolder, const fs::path& filename, const Json& options ) : - m_res(rootFolder), - m_cstamp( -1 ) +State::State( const fs::path& filename, const Json& options ) : + m_res ( options["path"].Str() ), + m_cstamp ( -1 ) { Read( filename ) ; diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index 2cc2405d..a0e12132 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -43,7 +43,7 @@ public : typedef ResourceTree::iterator iterator ; public : - explicit State( const fs::path& rootFolder, const fs::path& filename, const Json& options ) ; + explicit State( const fs::path& filename, const Json& options ) ; ~State() ; void FromLocal( const fs::path& p ) ; diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index 62a8333a..1a19effc 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -26,39 +26,80 @@ #include #include +namespace po = boost::program_options; + namespace gr { -Config::Config(const fs::path& configFile) : - m_configFile(configFile), - m_cfg( Read() ) +const std::string default_filename = ".grive"; +const char *env_name = "GR_CONFIG"; +const std::string default_root_folder = "."; + +Config::Config( const fs::path& root_path ) : + m_path( GetPath( root_path ) ) { - if (configFile.empty()) - { - throw Error() << expt::ErrMsg("Config cannot be initalised with an empty string."); - } + m_file = Read() ; +} + +Config::Config( const po::variables_map& vm ) +{ + m_cmd.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; + m_cmd.Add( "new-rev", Json(vm.count("new-rev") > 0) ) ; + m_cmd.Add( "force", Json(vm.count("force") > 0 ) ) ; + m_cmd.Add( "path", Json(vm.count("path") > 0 + ? vm["path"].as() + : default_root_folder ) ) ; + + m_path = GetPath( fs::path(m_cmd["path"].Str()) ) ; + m_file = Read( ) ; +} + +fs::path Config::GetPath( const fs::path& root_path ) +{ + // config file will be (in order of preference) + // value specified in environment string + // value specified in defaultConfigFileName in path from commandline --path + // value specified in defaultConfigFileName in current directory + const char *env = ::getenv( env_name ) ; + return root_path / (env ? env : default_filename) ; } -const fs::path& Config::ConfigFile() const +const fs::path Config::Filename() const { - return m_configFile ; + return m_path ; } void Config::Save( ) { - StdioFile file( m_configFile.string(), 0600 ) ; - m_cfg.Write( file ) ; + StdioFile file( m_path.string(), 0600 ) ; + m_file.Write( file ) ; +} + +void Config::Set( const std::string& key, const Json& value ) +{ + m_file.Add( key, value ) ; +} + +Json Config::Get( const std::string& key ) const +{ + return m_cmd.Has(key) ? m_cmd[key] : m_file[key] ; } -Json& Config::Get() +Json Config::GetAll() const { - return m_cfg ; + Json::Object obj = m_file.AsObject() ; + Json::Object cmd_obj = m_cmd.AsObject() ; + + for ( Json::Object::iterator i = cmd_obj.begin() ; i != cmd_obj.end() ; ++i ) + obj[i->first] = i->second ; + + return Json( obj ) ; } Json Config::Read() { try { - return Json::ParseFile( m_configFile.string() ) ; + return Json::ParseFile( m_path.string() ) ; } catch ( Exception& e ) { diff --git a/libgrive/src/util/Config.hh b/libgrive/src/util/Config.hh index 30ef116b..8349ce33 100644 --- a/libgrive/src/util/Config.hh +++ b/libgrive/src/util/Config.hh @@ -23,6 +23,14 @@ #include "FileSystem.hh" #include "protocol/Json.hh" +namespace boost +{ + namespace program_options + { + class variables_map ; + } +} + namespace gr { class Config @@ -31,18 +39,30 @@ public : struct Error : virtual Exception {} ; typedef boost::error_info File ; - Config(const fs::path& configFile) ; + Config( const fs::path& root_path ) ; + Config( const boost::program_options::variables_map& vm ) ; + + const fs::path Filename() const ; - Json& Get() ; + void Set( const std::string& key, const Json& value ) ; + Json Get( const std::string& key ) const ; + + Json GetAll() const ; void Save() ; - const fs::path &ConfigFile() const; private : - Json Read() ; + Json Read( ) ; + static fs::path GetPath( const fs::path& root_path ) ; private : - const fs::path m_configFile; - Json m_cfg ; + //! config file path + fs::path m_path; + + //! config values loaded from config file + Json m_file ; + + //! config values from command line + Json m_cmd ; } ; } // end of namespace diff --git a/libgrive/test/util/ConfigTest.cc b/libgrive/test/util/ConfigTest.cc index 33a427d9..94b3eeff 100644 --- a/libgrive/test/util/ConfigTest.cc +++ b/libgrive/test/util/ConfigTest.cc @@ -36,22 +36,22 @@ ConfigTest::ConfigTest( ) void ConfigTest::TestInitialiseWithEmptyString( ) { - Config config(""); - GRUT_ASSERT_EQUAL( "/home/.grive", config.ConfigFile().string()) ; + Config config(""); + GRUT_ASSERT_EQUAL( "/home/.grive", config.Filename().string()) ; } void ConfigTest::TestInitialiseWithString( ) { - Config config("/home/.grive"); - GRUT_ASSERT_EQUAL( "/home/.grive", config.ConfigFile().string()) ; + Config config("/home/.grive"); + GRUT_ASSERT_EQUAL( "/home/.grive", config.Filename().string()) ; } void ConfigTest::TestInitialiseWithFileSystemPath( ) { - fs::path path("/home"); - fs::path file(".grive"); - Config config(path / file); - GRUT_ASSERT_EQUAL( "/home/.grive", config.ConfigFile().string()); + fs::path path("/home"); + fs::path file(".grive"); + Config config(path / file); + GRUT_ASSERT_EQUAL( "/home/.grive", config.Filename().string()); } From 29f17397c18c9e7eff22e4d52d9c388393ab3e38 Mon Sep 17 00:00:00 2001 From: justin Date: Wed, 1 Aug 2012 20:46:59 +0100 Subject: [PATCH 031/166] fix broken unit tests --- libgrive/src/util/Config.cc | 6 ------ libgrive/src/util/Config.hh | 1 - libgrive/test/util/ConfigTest.cc | 36 +++++++++++++++++++------------- libgrive/test/util/ConfigTest.hh | 5 ++--- 4 files changed, 24 insertions(+), 24 deletions(-) diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index 1a19effc..539e82c0 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -34,12 +34,6 @@ const std::string default_filename = ".grive"; const char *env_name = "GR_CONFIG"; const std::string default_root_folder = "."; -Config::Config( const fs::path& root_path ) : - m_path( GetPath( root_path ) ) -{ - m_file = Read() ; -} - Config::Config( const po::variables_map& vm ) { m_cmd.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; diff --git a/libgrive/src/util/Config.hh b/libgrive/src/util/Config.hh index 8349ce33..e2c0423f 100644 --- a/libgrive/src/util/Config.hh +++ b/libgrive/src/util/Config.hh @@ -39,7 +39,6 @@ public : struct Error : virtual Exception {} ; typedef boost::error_info File ; - Config( const fs::path& root_path ) ; Config( const boost::program_options::variables_map& vm ) ; const fs::path Filename() const ; diff --git a/libgrive/test/util/ConfigTest.cc b/libgrive/test/util/ConfigTest.cc index 94b3eeff..48d7602c 100644 --- a/libgrive/test/util/ConfigTest.cc +++ b/libgrive/test/util/ConfigTest.cc @@ -25,33 +25,41 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. #include "protocol/Json.hh" #include "util/log/Log.hh" +#include #include using namespace grut; using namespace gr ; +namespace po = boost::program_options; ConfigTest::ConfigTest( ) { } -void ConfigTest::TestInitialiseWithEmptyString( ) +void ConfigTest::TestInitialiseWithNoPath( ) { - Config config(""); - GRUT_ASSERT_EQUAL( "/home/.grive", config.Filename().string()) ; -} + po::variables_map vm; + po::notify(vm); -void ConfigTest::TestInitialiseWithString( ) -{ - Config config("/home/.grive"); - GRUT_ASSERT_EQUAL( "/home/.grive", config.Filename().string()) ; + Config config(vm); + GRUT_ASSERT_EQUAL( "./.grive", config.Filename().string()) ; } -void ConfigTest::TestInitialiseWithFileSystemPath( ) +void ConfigTest::TestInitialiseWithPath( ) { - fs::path path("/home"); - fs::path file(".grive"); - Config config(path / file); - GRUT_ASSERT_EQUAL( "/home/.grive", config.Filename().string()); -} + char const *argv[] = { "Program", "-p", "/home/grive" }; + int argc = 3; + po::options_description desc( "Grive options" ); + desc.add_options() + ( "path,p", po::value(), "Path to sync") + ; + + po::variables_map vm; + po::store(po::parse_command_line( argc, argv, desc), vm ); + po::notify(vm); + + Config config(vm); + GRUT_ASSERT_EQUAL( "/home/grive/.grive", config.Filename().string()) ; +} diff --git a/libgrive/test/util/ConfigTest.hh b/libgrive/test/util/ConfigTest.hh index 90325743..f7e100d3 100644 --- a/libgrive/test/util/ConfigTest.hh +++ b/libgrive/test/util/ConfigTest.hh @@ -30,12 +30,11 @@ class ConfigTest : public CppUnit::TestFixture public : ConfigTest( ) ; - // declare suit function CPPUNIT_TEST_SUITE( ConfigTest ) ; - CPPUNIT_TEST_EXCEPTION( TestInitialiseWithEmptyString, gr::Config::Error ) ; + CPPUNIT_TEST( TestInitialiseWithEmptyString ) ; CPPUNIT_TEST( TestInitialiseWithString ) ; CPPUNIT_TEST( TestInitialiseWithFileSystemPath ) ; - CPPUNIT_TEST_SUITE_END(); + CPPUNIT_TEST_SUITE_END(); private : void TestInitialiseWithEmptyString( ); From 755ee1c9adfe290f68e18ddea0b01673186b3917 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Sun, 28 Apr 2013 18:40:21 +0800 Subject: [PATCH 032/166] new qt gui --- CMakeLists.txt | 2 + bgrive/CMakeLists.txt | 43 ++++ bgrive/src/MainWnd.cc | 30 +++ bgrive/src/MainWnd.hh | 40 ++++ bgrive/src/main.cc | 78 ++++++ bgrive/ui/MainWindow.ui | 86 +++++++ libgrive/src/drive/Feed.hh | 2 +- libgrive/src/drive/Resource.cc | 6 +- libgrive/src/drive/State.cc | 6 +- libgrive/src/http/Agent.hh | 6 +- libgrive/src/http/CurlAgent.cc | 8 +- libgrive/src/http/CurlAgent.hh | 2 +- libgrive/src/http/Download.cc | 2 +- libgrive/src/http/Download.hh | 4 +- libgrive/src/http/MarshalAgent.hh | 39 +++ libgrive/src/http/Receivable.hh | 1 + libgrive/src/protocol/AuthAgent.cc | 2 +- libgrive/src/protocol/AuthAgent.hh | 2 +- libgrive/src/protocol/Json.cc | 248 ++++++++++++++++---- libgrive/src/protocol/Json.hh | 49 +++- libgrive/src/util/Config.cc | 11 +- libgrive/src/util/Config.hh | 2 +- libgrive/src/util/Crypt.cc | 8 +- libgrive/src/util/Crypt.hh | 4 +- libgrive/src/util/DataStream.hh | 55 +++++ libgrive/src/util/Exception.cc | 11 +- libgrive/src/util/Exception.hh | 43 +--- libgrive/src/util/{StdioFile.cc => File.cc} | 107 ++++++--- libgrive/src/util/{StdioFile.hh => File.hh} | 33 ++- libgrive/src/util/FileSystem.hh | 13 - libgrive/src/util/MemMap.cc | 6 +- libgrive/src/util/MemMap.hh | 4 +- 32 files changed, 777 insertions(+), 176 deletions(-) create mode 100644 bgrive/CMakeLists.txt create mode 100644 bgrive/src/MainWnd.cc create mode 100644 bgrive/src/MainWnd.hh create mode 100644 bgrive/src/main.cc create mode 100644 bgrive/ui/MainWindow.ui create mode 100644 libgrive/src/http/MarshalAgent.hh create mode 100644 libgrive/src/util/DataStream.hh rename libgrive/src/util/{StdioFile.cc => File.cc} (63%) rename libgrive/src/util/{StdioFile.hh => File.hh} (66%) diff --git a/CMakeLists.txt b/CMakeLists.txt index def7ca00..7d16f2de 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -6,6 +6,8 @@ set( GRIVE_VERSION "0.3.0-pre" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) add_definitions( -D_FILE_OFFSET_BITS=64 ) +add_definitions( -DPROJ_NS=gr ) add_subdirectory( libgrive ) add_subdirectory( grive ) +add_subdirectory( bgrive ) diff --git a/bgrive/CMakeLists.txt b/bgrive/CMakeLists.txt new file mode 100644 index 00000000..074ab383 --- /dev/null +++ b/bgrive/CMakeLists.txt @@ -0,0 +1,43 @@ +project( bgrive ) + +find_package(Qt4 REQUIRED) +find_package(Boost REQUIRED) +INCLUDE(${QT_USE_FILE}) + +include_directories( + ${bgrive_SOURCE_DIR}/../libgrive/src + ${CMAKE_CURRENT_SOURCE_DIR} + ${CMAKE_CURRENT_BINARY_DIR} +) + +file (GLOB BGRIVE_EXE_SRC + ${bgrive_SOURCE_DIR}/src/*.cc +) + +file (GLOB BGRIVE_UI + ${bgrive_SOURCE_DIR}/ui/*.ui +) + +QT4_WRAP_UI(BGRIVE_UI_SRCS ${BGRIVE_UI}) +QT4_WRAP_CPP(BGRIVE_MOC_SRCS + src/MainWnd.hh ) + +add_executable( bgrive_executable + ${BGRIVE_EXE_SRC} + ${BGRIVE_UI_SRCS} + ${BGRIVE_MOC_SRCS} +) + +target_link_libraries( bgrive_executable + ${Boost_LIBRARIES} + ${QT_QTMAIN_LIBRARY} + ${QT_LIBRARIES} + grive +) + +set_target_properties( bgrive_executable + PROPERTIES OUTPUT_NAME bgrive +) + +install(TARGETS bgrive_executable RUNTIME DESTINATION bin) +install(FILES doc/grive.1 DESTINATION share/man/man1 ) diff --git a/bgrive/src/MainWnd.cc b/bgrive/src/MainWnd.cc new file mode 100644 index 00000000..1712ebc8 --- /dev/null +++ b/bgrive/src/MainWnd.cc @@ -0,0 +1,30 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "MainWnd.hh" + +namespace gr { + +MainWnd::MainWnd( ) +{ + m_ui.setupUi(this) ; +} + +} // end of namespace diff --git a/bgrive/src/MainWnd.hh b/bgrive/src/MainWnd.hh new file mode 100644 index 00000000..f50d431e --- /dev/null +++ b/bgrive/src/MainWnd.hh @@ -0,0 +1,40 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include +#include "ui_MainWindow.h" + +namespace gr { + +class MainWnd : public QMainWindow +{ + Q_OBJECT + +public : + MainWnd( ) ; + +private : + Ui::MainWindow m_ui ; +} ; + +} // end of namespace + diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc new file mode 100644 index 00000000..4b61317b --- /dev/null +++ b/bgrive/src/main.cc @@ -0,0 +1,78 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "MainWnd.hh" + +#include +#include + +#include "drive/CommonUri.hh" +#include "drive/Entry.hh" +#include "drive/Feed.hh" + +#include "http/CurlAgent.hh" +#include "http/Header.hh" +#include "http/XmlResponse.hh" + +#include "protocol/Json.hh" +#include "protocol/OAuth2.hh" +#include "protocol/AuthAgent.hh" + +#include "util/File.hh" + +#include + +const std::string client_id = "22314510474.apps.googleusercontent.com" ; +const std::string client_secret = "bl4ufi89h-9MkFlypcI7R785" ; + +using namespace gr ; + +int main( int argc, char **argv ) +{ + File file( ".grive" ) ; + Json cfg = Json::Parse( &file ) ; + + std::string refresh_token = cfg["refresh_token"].Str() ; + qDebug() << refresh_token.c_str() ; + + OAuth2 token( refresh_token, client_id, client_secret ) ; + AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; + + + http::XmlResponse xml ; + agent.Get( feed_base + "/-/folder?max-results=50&showroot=true", &xml, http::Header() ) ; + + Feed feed( xml.Response() ) ; + do + { + // first, get all collections from the query result + for ( Feed::iterator i = feed.begin() ; i != feed.end() ; ++i ) + { + Entry e( *i ) ; + qDebug() << e.Name().c_str() ; + } + } while ( feed.GetNext( &agent, http::Header() ) ) ; + + QApplication app( argc, argv ) ; + MainWnd wnd ; + wnd.show(); + + return app.exec() ; +} diff --git a/bgrive/ui/MainWindow.ui b/bgrive/ui/MainWindow.ui new file mode 100644 index 00000000..e69c6006 --- /dev/null +++ b/bgrive/ui/MainWindow.ui @@ -0,0 +1,86 @@ + + + MainWindow + + + + 0 + 0 + 800 + 600 + + + + Grive + + + + + + + Qt::Horizontal + + + + + 0 + 0 + + + + + + + 1 + 0 + + + + + + + + + + + 0 + 0 + 800 + 23 + + + + + &File + + + + + + + + + E&xit + + + + + + + m_action_exit + activated() + MainWindow + close() + + + -1 + -1 + + + 399 + 299 + + + + + diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/drive/Feed.hh index f41fb7d3..deaf18c1 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/drive/Feed.hh @@ -67,7 +67,7 @@ private : std::auto_ptr m_log ; xml::Node m_root ; - xml::NodeSet m_entries ; + xml::NodeSet m_entries ; } ; class Feed::iterator : public boost::iterator_adaptor< diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index e64bd0fe..63a54bd6 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -32,7 +32,7 @@ #include "util/Crypt.hh" #include "util/log/Log.hh" #include "util/OS.hh" -#include "util/StdioFile.hh" +#include "util/File.hh" #include "xml/Node.hh" #include "xml/NodeSet.hh" #include "xml/String.hh" @@ -269,7 +269,7 @@ void Resource::FromLocal( const DateTime& last_sync ) else m_state = ( m_mtime > last_sync ? local_new : remote_deleted ) ; - m_name = Path2Str( path.filename() ) ; + m_name = path.filename().string() ; m_kind = fs::is_directory(path) ? "folder" : "file" ; m_md5 = fs::is_directory(path) ? "" : crypt::MD5::Get( path ) ; } @@ -583,7 +583,7 @@ bool Resource::Upload( { assert( http != 0 ) ; - StdioFile file( Path() ) ; + File file( Path() ) ; std::ostringstream xcontent_len ; xcontent_len << "X-Upload-Content-Length: " << file.Size() ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 43e63679..195e6967 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -25,6 +25,7 @@ #include "http/Agent.hh" #include "util/Crypt.hh" +#include "util/File.hh" #include "util/log/Log.hh" #include "protocol/Json.hh" @@ -72,7 +73,7 @@ void State::FromLocal( const fs::path& p, gr::Resource* folder ) for ( fs::directory_iterator i( p ) ; i != fs::directory_iterator() ; ++i ) { - std::string fname = Path2Str(i->path().filename()) ; + std::string fname = i->path().filename().string() ; if ( IsIgnore(fname) ) Log( "file %1% is ignored by grive", fname, log::verbose ) ; @@ -228,7 +229,8 @@ void State::Read( const fs::path& filename ) { try { - Json json = Json::ParseFile( filename.string() ) ; + File file( filename ) ; + Json json = Json::Parse( &file ) ; Json last_sync = json["last_sync"] ; m_last_sync.Assign( diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index 0baa0320..0ca1e9b7 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -23,7 +23,7 @@ namespace gr { -class StdioFile ; +class File ; namespace http { @@ -33,6 +33,8 @@ class Receivable ; class Agent { public : + virtual ~Agent() {} + virtual long Put( const std::string& url, const std::string& data, @@ -41,7 +43,7 @@ public : virtual long Put( const std::string& url, - StdioFile& file, + File& file, Receivable *dest, const Header& hdr ) = 0 ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 09bedee5..2cf78b81 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -25,7 +25,7 @@ #include "Receivable.hh" #include "util/log/Log.hh" -#include "util/StdioFile.hh" +#include "util/File.hh" #include @@ -62,7 +62,7 @@ size_t ReadStringCallback( void *ptr, std::size_t size, std::size_t nmemb, std:: return count ; } -size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, StdioFile *file ) +size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, File *file ) { assert( ptr != 0 ) ; assert( file != 0 ) ; @@ -73,7 +73,7 @@ size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, StdioFi assert( count <= std::numeric_limits::max() ) ; if ( count > 0 ) - file->Read( ptr, static_cast(count) ) ; + file->Read( static_cast(ptr), static_cast(count) ) ; return count ; } @@ -197,7 +197,7 @@ long CurlAgent::Put( long CurlAgent::Put( const std::string& url, - StdioFile& file, + File& file, Receivable *dest, const Header& hdr ) { diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index 9db1d23b..2b9a394a 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -47,7 +47,7 @@ public : long Put( const std::string& url, - StdioFile& file, + File& file, Receivable *dest, const Header& hdr ) ; diff --git a/libgrive/src/http/Download.cc b/libgrive/src/http/Download.cc index 15171cdf..3b98ac4d 100644 --- a/libgrive/src/http/Download.cc +++ b/libgrive/src/http/Download.cc @@ -72,7 +72,7 @@ std::size_t Download::OnData( void *data, std::size_t count ) if ( m_crypt.get() != 0 ) m_crypt->Write( data, count ) ; - return m_file.Write( data, count ) ; + return m_file.Write( static_cast(data), count ) ; } } } // end of namespace diff --git a/libgrive/src/http/Download.hh b/libgrive/src/http/Download.hh index d3bdc59d..683f864a 100644 --- a/libgrive/src/http/Download.hh +++ b/libgrive/src/http/Download.hh @@ -20,7 +20,7 @@ #pragma once #include "Receivable.hh" -#include "util/StdioFile.hh" +#include "util/File.hh" #include @@ -47,7 +47,7 @@ public : std::size_t OnData( void *data, std::size_t count ) ; private : - StdioFile m_file ; + File m_file ; std::auto_ptr m_crypt ; } ; diff --git a/libgrive/src/http/MarshalAgent.hh b/libgrive/src/http/MarshalAgent.hh new file mode 100644 index 00000000..0ca2b272 --- /dev/null +++ b/libgrive/src/http/MarshalAgent.hh @@ -0,0 +1,39 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "http/Agent.hh" + +namespace gr { namespace http { + +class HttpRequest ; + +/*! \brief An marshaller for HTTP agent + + This agent will marshal the HTTP requests to a request objects containing the arguments + of the request and the response. +*/ +class MarshalAgent : public Agent +{ +public : + MarshalAgent( std::auto_ptr real_agent ) ; +} ; + +}} // end of namespace diff --git a/libgrive/src/http/Receivable.hh b/libgrive/src/http/Receivable.hh index 2982aa1a..0eb1afb5 100644 --- a/libgrive/src/http/Receivable.hh +++ b/libgrive/src/http/Receivable.hh @@ -26,6 +26,7 @@ namespace gr { namespace http { class Receivable { public : + virtual ~Receivable() {} virtual std::size_t OnData( void *data, std::size_t count ) = 0 ; virtual void Clear() = 0 ; } ; diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 9d0239b8..c3cdabb7 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -62,7 +62,7 @@ long AuthAgent::Put( long AuthAgent::Put( const std::string& url, - StdioFile& file, + File& file, Receivable *dest, const Header& hdr ) { diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index 58900d18..fe6ae22a 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -44,7 +44,7 @@ public : long Put( const std::string& url, - StdioFile& file, + File& file, http::Receivable *dest, const http::Header& hdr ) ; diff --git a/libgrive/src/protocol/Json.cc b/libgrive/src/protocol/Json.cc index 7012db77..58da219b 100644 --- a/libgrive/src/protocol/Json.cc +++ b/libgrive/src/protocol/Json.cc @@ -19,10 +19,21 @@ #include "Json.hh" -#include "util/StdioFile.hh" +#include "util/DataStream.hh" +// needs to include stdint.h before json-c to avoid macro re-def warning +#include + +// disable macro re-def warning for json-c headers +#ifdef _MSC_VER + #pragma warning(push) + #pragma warning(disable: 4005) +#endif #include #include +#ifdef _MSC_VER + #pragma warning(pop) +#endif #include #include @@ -35,51 +46,99 @@ Json::Json( ) : m_json( ::json_object_new_object() ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json object" ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_object" ) + ) ; } Json::Json( const char *str ) : m_json( ::json_object_new_string( str ) ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() << expt::ErrMsg( "cannot create json string \"" + std::string(str) + "\"" ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_string" ) + << ValueErr( str ) + ) ; +} + +struct json_object* Json::InitStr( const char *str, std::size_t n ) +{ + struct json_object *j = ::json_object_new_string_len( str, n ) ; + if ( j == 0 ) + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_string_len" ) + << ValueErr( std::string(str, n) ) + ) ; + return j ; } template <> Json::Json( const std::string& str ) : - m_json( ::json_object_new_string( str.c_str() ) ) + m_json( InitStr( str.c_str(), str.size() ) ) { - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() << expt::ErrMsg( "cannot create json string \"" + str + "\"" ) ) ; +} - // paranoid check - assert( ::json_object_get_string( m_json ) == str ) ; +template <> +Json::Json( const double& val ) : + m_json( ::json_object_new_double( val ) ) +{ + if ( m_json == 0 ) + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_double" ) + << ValueErr( val ) + ) ; } template <> -Json::Json( const int& l ) : +Json::Json( const boost::int32_t& l ) : m_json( ::json_object_new_int( l ) ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json int" ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_int" ) + << ValueErr( l ) + ) ; } template <> -Json::Json( const long& l ) : - m_json( ::json_object_new_int( static_cast(l) ) ) +Json::Json( const boost::int64_t& l ) : + m_json( ::json_object_new_int64( l ) ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json int" ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_int64" ) + << ValueErr( l ) + ) ; } template <> -Json::Json( const unsigned long& l ) : +Json::Json( const boost::uint32_t& l ) : m_json( ::json_object_new_int( static_cast(l) ) ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json int" ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_int" ) + << ValueErr( l ) + ) ; +} + +template <> +Json::Json( const boost::uint64_t& l ) : + m_json( ::json_object_new_int64( l ) ) +{ + if ( m_json == 0 ) + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_int64" ) + << ValueErr( l ) + ) ; } template <> @@ -87,7 +146,7 @@ Json::Json( const std::vector& arr ) : m_json( ::json_object_new_array( ) ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json int" ) ) ; + BOOST_THROW_EXCEPTION( Error() << JsonCApi_( "json_object_new_array" ) ) ; for ( std::vector::const_iterator i = arr.begin() ; i != arr.end() ; ++i ) Add( *i ) ; @@ -98,7 +157,11 @@ Json::Json( const bool& b ) : m_json( ::json_object_new_boolean( b ) ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json bool" ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_object_new_boolean" ) + << ValueErr( b ) + ) ; } template <> @@ -106,7 +169,7 @@ Json::Json( const Object& obj ) : m_json( ::json_object_new_object() ) { if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "cannot create json object" ) ) ; + BOOST_THROW_EXCEPTION( Error() << JsonCApi_( "json_object_new_object" ) ) ; for ( Object::const_iterator i = obj.begin() ; i != obj.end() ; ++i ) Add( i->first, i->second ) ; @@ -157,13 +220,15 @@ Json Json::operator[]( const std::string& key ) const { assert( m_json != 0 ) ; - struct json_object *j = ::json_object_object_get( m_json, key.c_str() ) ; - if ( j == 0 ) + struct json_object *j = 0 ; + if ( !::json_object_object_get_ex( m_json, key.c_str(), &j ) ) BOOST_THROW_EXCEPTION( Error() - << expt::ErrMsg( "key: " + key + " is not found in object" ) - << JsonInfo( *this ) ) ; + << JsonCApi_( "json_object_object_get" ) + << KeyNotFound_( key ) + << Json_( ::json_object_to_json_string(m_json) ) ) ; + assert( j != 0 ) ; return Json( j ) ; } @@ -174,12 +239,11 @@ Json Json::operator[]( const std::size_t& idx ) const struct json_object *j = ::json_object_array_get_idx( m_json, idx ) ; if ( j == 0 ) { - std::ostringstream ss ; - ss << "index " << idx << " is not found in array" ; BOOST_THROW_EXCEPTION( Error() - << expt::ErrMsg( ss.str() ) - << JsonInfo( *this ) ) ; + << JsonCApi_( "json_object_array_get_idx" ) + << OutOfRange_( idx ) + << Json_( ::json_object_to_json_string(m_json) ) ) ; } return Json( j ) ; @@ -188,15 +252,17 @@ Json Json::operator[]( const std::size_t& idx ) const bool Json::Has( const std::string& key ) const { assert( m_json != 0 ) ; - return ::json_object_object_get( m_json, key.c_str() ) != 0 ; + return ::json_object_object_get_ex( m_json, key.c_str(), 0 ) == TRUE ; } bool Json::Get( const std::string& key, Json& json ) const { assert( m_json != 0 ) ; - struct json_object *j = ::json_object_object_get( m_json, key.c_str() ) ; - if ( j != 0 ) + struct json_object *j = 0 ; + if ( ::json_object_object_get_ex( m_json, key.c_str(), &j ) ) { + assert( j != 0 ) ; + Json tmp( j ) ; json.Swap( tmp ) ; return true ; @@ -226,14 +292,20 @@ void Json::Add( const Json& json ) bool Json::Bool() const { assert( m_json != 0 ) ; - return ::json_object_get_boolean( m_json ) ; + return ::json_object_get_boolean( m_json ) == TRUE ; } template <> bool Json::Is() const { assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_boolean ) ; + return ::json_object_is_type( m_json, json_type_boolean ) == TRUE ; +} + +template <> +bool Json::As() const +{ + return Bool() ; } std::string Json::Str() const @@ -246,7 +318,13 @@ template <> bool Json::Is() const { assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_string ) ; + return ::json_object_is_type( m_json, json_type_string ) == TRUE ; +} + +template <> +std::string Json::As() const +{ + return Str() ; } int Json::Int() const @@ -259,7 +337,31 @@ template <> bool Json::Is() const { assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_int ) ; + return ::json_object_is_type( m_json, json_type_int ) == TRUE ; +} + +template <> +boost::int32_t Json::As() const +{ + return Int() ; +} + +template <> +boost::uint32_t Json::As() const +{ + return static_cast(Int()) ; +} + +template <> +boost::int64_t Json::As() const +{ + return ::json_object_get_int64( m_json ) ; +} + +template <> +boost::uint64_t Json::As() const +{ + return ::json_object_get_int64( m_json ) ; } std::ostream& operator<<( std::ostream& os, const Json& json ) @@ -268,10 +370,12 @@ std::ostream& operator<<( std::ostream& os, const Json& json ) return os << ::json_object_to_json_string( json.m_json ) ; } -void Json::Write( StdioFile& file ) const +void Json::Write( DataStream *out ) const { + assert( out != 0 ) ; + const char *str = ::json_object_to_json_string( m_json ) ; - file.Write( str, std::strlen(str) ) ; + out->Write( str, std::strlen(str) ) ; } Json::Type Json::DataType() const @@ -296,7 +400,13 @@ template <> bool Json::Is() const { assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_object ) ; + return ::json_object_is_type( m_json, json_type_object ) == TRUE ; +} + +template <> +Json::Object Json::As() const +{ + return AsObject() ; } Json::Array Json::AsArray() const @@ -314,9 +424,18 @@ template <> bool Json::Is() const { assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_array ) ; + return ::json_object_is_type( m_json, json_type_array ) == TRUE ; +} + +template <> +Json::Array Json::As() const +{ + return AsArray() ; } +/// Finds an element in the array. +/// \pre "this" is an array +/// \return *this[i] if *this[i][key] == value Json Json::FindInArray( const std::string& key, const std::string& value ) const { std::size_t count = ::json_object_array_length( m_json ) ; @@ -327,8 +446,16 @@ Json Json::FindInArray( const std::string& key, const std::string& value ) const if ( item.Has(key) && item[key].Str() == value ) return item ; } + BOOST_THROW_EXCEPTION( - Error() << expt::ErrMsg( "cannot find " + key + " = " + value + " in array" ) ) ; + Error() + << JsonCApi_( "Json::FindInArray" ) + << KeyNotFound_( key ) + << Value_(value) + ) ; + + // shut off compiler warnings + return Json() ; } bool Json::FindInArray( const std::string& key, const std::string& value, Json& result ) const @@ -340,37 +467,58 @@ bool Json::FindInArray( const std::string& key, const std::string& value, Json& } catch ( Error& ) { - return false ; } + return false ; } Json Json::Parse( const std::string& str ) { struct json_object *json = ::json_tokener_parse( str.c_str() ) ; if ( json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "json parse error" ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_tokener_parse" ) + << ValueErr( str ) + ) ; return Json( json, NotOwned() ) ; } -Json Json::ParseFile( const std::string& filename ) +/// Parse a file. The file is loaded from file system. +/// \throw Error expt::ErrMsg contains a human-readable message describing the +/// error. +Json Json::Parse( DataStream *in ) { - StdioFile file( filename ) ; + assert( in != 0 ) ; + struct json_tokener *tok = ::json_tokener_new() ; - struct json_object *json = 0 ; char buf[1024] ; std::size_t count = 0 ; - while ( (count = file.Read( buf, sizeof(buf) ) ) > 0 ) + while ( (count = in->Read( buf, sizeof(buf) ) ) > 0 ) + { json = ::json_tokener_parse_ex( tok, buf, count ) ; + + // check for parse error + if ( ::json_tokener_get_error(tok) == ::json_tokener_continue ) + break ; + } - if ( json == 0 ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( ::json_tokener_errors[tok->err] ) ) ; - - ::json_tokener_free( tok ) ; + // save the error code and free the tokener before throwing exceptions + ::json_tokener_error err = ::json_tokener_get_error(tok) ; + ::json_tokener_free( tok ) ; tok = 0 ; + if ( err != json_tokener_success || json == 0 ) + { + BOOST_THROW_EXCEPTION( + Error() + << JsonCApi_( "json_tokener_parse" ) + << expt::ErrMsg( ::json_tokener_error_desc(err) ) + ) ; + } + return Json( json, NotOwned() ) ; } diff --git a/libgrive/src/protocol/Json.hh b/libgrive/src/protocol/Json.hh index 034a4fc7..a1f6bf0e 100644 --- a/libgrive/src/protocol/Json.hh +++ b/libgrive/src/protocol/Json.hh @@ -29,8 +29,13 @@ struct json_object ; namespace gr { -class StdioFile ; +class DataStream ; +/*! \brief Simple wrapper around JSON-C objects. + + This class represents JSON-C objects, which can be integers, booleans, strings + double, arrays and object. +*/ class Json { public : @@ -38,19 +43,35 @@ public : typedef std::vector Array ; struct Error : virtual Exception {} ; - typedef boost::error_info JsonInfo ; - + typedef boost::error_info Json_ ; + typedef boost::error_info OutOfRange_ ; + typedef boost::error_info KeyNotFound_ ; + typedef boost::error_info JsonCApi_ ; + typedef boost::error_info Value_ ; + + template + struct Val_ + { + typedef boost::error_info Err ; + } ; + public : template explicit Json( const T& val ) ; + template + explicit Json( const char (&str)[n] ) : + m_json( InitStr( str, n ) ) + { + } + Json() ; Json( const Json& rhs ) ; Json( const char *str ) ; - ~Json( ) ; + ~Json() ; static Json Parse( const std::string& str ) ; - static Json ParseFile( const std::string& filename ) ; + static Json Parse( DataStream *in ) ; Json operator[]( const std::string& key ) const ; Json operator[]( const std::size_t& idx ) const ; @@ -68,6 +89,9 @@ public : template bool Is() const ; + template + T As() const ; + bool Has( const std::string& key ) const ; bool Get( const std::string& key, Json& json ) const ; void Add( const std::string& key, const Json& json ) ; @@ -76,7 +100,7 @@ public : bool FindInArray( const std::string& key, const std::string& value, Json& result ) const ; friend std::ostream& operator<<( std::ostream& os, const Json& json ) ; - void Write( StdioFile& file ) const ; + void Write( DataStream *out ) const ; enum Type { null_type, bool_type, double_type, int_type, object_type, array_type, string_type } ; @@ -87,10 +111,17 @@ private : struct NotOwned {} ; Json( struct json_object *json, NotOwned ) ; - + + static struct json_object* InitStr( const char *str, std::size_t n ) ; + + // helper for throwing exception + template static typename Val_::Err ValueErr( const T& t ) + { + return typename Val_::Err(t); + } + private : -public : struct json_object *m_json ; } ; -} \ No newline at end of file +} diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index 539e82c0..6111b1d4 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -19,7 +19,7 @@ #include "Config.hh" -#include "util/StdioFile.hh" +#include "util/File.hh" #include @@ -28,7 +28,7 @@ namespace po = boost::program_options; -namespace gr { +namespace PROJ_NS { const std::string default_filename = ".grive"; const char *env_name = "GR_CONFIG"; @@ -64,8 +64,8 @@ const fs::path Config::Filename() const void Config::Save( ) { - StdioFile file( m_path.string(), 0600 ) ; - m_file.Write( file ) ; + gr::File file( m_path.string(), 0600 ) ; + m_file.Write( &file ) ; } void Config::Set( const std::string& key, const Json& value ) @@ -93,7 +93,8 @@ Json Config::Read() { try { - return Json::ParseFile( m_path.string() ) ; + gr::File file(m_path) ; + return Json::Parse( &file ) ; } catch ( Exception& e ) { diff --git a/libgrive/src/util/Config.hh b/libgrive/src/util/Config.hh index e2c0423f..6057f148 100644 --- a/libgrive/src/util/Config.hh +++ b/libgrive/src/util/Config.hh @@ -31,7 +31,7 @@ namespace boost } } -namespace gr { +namespace PROJ_NS { class Config { diff --git a/libgrive/src/util/Crypt.cc b/libgrive/src/util/Crypt.cc index 953767d3..f0049f57 100644 --- a/libgrive/src/util/Crypt.cc +++ b/libgrive/src/util/Crypt.cc @@ -19,7 +19,7 @@ #include "Crypt.hh" -#include "StdioFile.hh" +#include "File.hh" #include "Exception.hh" #include "MemMap.hh" @@ -76,16 +76,16 @@ std::string MD5::Get( const fs::path& file ) { try { - StdioFile sfile( file ) ; + File sfile( file ) ; return Get( sfile ) ; } - catch ( StdioFile::Error& ) + catch ( File::Error& ) { return "" ; } } -std::string MD5::Get( StdioFile& file ) +std::string MD5::Get( File& file ) { MD5 crypt ; diff --git a/libgrive/src/util/Crypt.hh b/libgrive/src/util/Crypt.hh index 6126b53c..6cd6ceba 100644 --- a/libgrive/src/util/Crypt.hh +++ b/libgrive/src/util/Crypt.hh @@ -26,7 +26,7 @@ namespace gr { -class StdioFile ; +class File ; namespace crypt { @@ -36,7 +36,7 @@ public : MD5() ; ~MD5() ; - static std::string Get( StdioFile& file ) ; + static std::string Get( File& file ) ; static std::string Get( const boost::filesystem::path& file ) ; void Write( const void *data, std::size_t size ) ; diff --git a/libgrive/src/util/DataStream.hh b/libgrive/src/util/DataStream.hh new file mode 100644 index 00000000..819c8f8a --- /dev/null +++ b/libgrive/src/util/DataStream.hh @@ -0,0 +1,55 @@ +/* + webwrite: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include + +namespace gr { + +/** \brief Encapsulation of data streams. Useful for unit tests. + This class provides two functions: Read() and Write(). +*/ +class DataStream +{ +protected : + virtual ~DataStream() {} + +public : + /** Reading from the stream. The caller indicates that it wants + to read `size` bytes and must provide enough space pointed + by `data`. + \param data Buffer to hold the data read from the stream + Must have at least `size` bytes. + \param size Number of bytes the caller wants to read. + \throw wb::Exception In case of any error. + \return The number of byte actually read from the stream. + 0 indicates the end of stream, i.e. you will + still get 0 if you call again. + */ + virtual std::size_t Read( char *data, std::size_t size ) = 0 ; + virtual std::size_t Write( const char *data, std::size_t size ) = 0 ; +} ; + +/// Stream for /dev/null, i.e. read and writing nothing +DataStream* DevNull() ; + + + +} // end of namespace diff --git a/libgrive/src/util/Exception.cc b/libgrive/src/util/Exception.cc index 68738b44..a410bbb5 100644 --- a/libgrive/src/util/Exception.cc +++ b/libgrive/src/util/Exception.cc @@ -17,7 +17,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "util/Exception.hh" +#include "Exception.hh" #include "bfd/Backtrace.hh" #include "bfd/Debug.hh" @@ -28,15 +28,20 @@ #include #include -namespace gr { +namespace PROJ_NS { class Backtrace ; Exception::Exception( ) { #ifdef HAVE_BFD - *this << expt::BacktraceInfo( Backtrace() ) ; + *this << expt::Backtrace_( Backtrace() ) ; #endif } +const char* Exception::what() const throw() +{ + return boost::diagnostic_information_what( *this ) ; +} + } // end of namespace diff --git a/libgrive/src/util/Exception.hh b/libgrive/src/util/Exception.hh index fec5af71..164f9dd5 100644 --- a/libgrive/src/util/Exception.hh +++ b/libgrive/src/util/Exception.hh @@ -1,5 +1,5 @@ /* - grive: an GPL program to sync a local directory with Google Drive + webwrite: an GPL program to sync a local directory with Google Drive Copyright (C) 2012 Wan Wai Ho This program is free software; you can redistribute it and/or @@ -25,54 +25,35 @@ #include #include -namespace gr { +namespace PROJ_NS { class Backtrace ; /** \defgroup exception Exception Classes */ -/// base class for exception in libpdfdoc -/** \ingroup exception - This class is the base class for all exception class in libpdfdoc. +/** \brief base class for exception in WebWrite + \ingroup exception + This class is the base class for all exception class in WebWrite. + It allows us to catch all WebWrite exception with one catch clause. */ struct Exception : virtual public std::exception, virtual public boost::exception { Exception( ) ; + + virtual const char* what() const throw() ; } ; -struct FileError : virtual Exception {} ; - -/// Parse error exception. -/** \ingroup exception - This exception will be thrown when there is a parse error when reading - a PDF file. -*/ -struct ParseError : virtual Exception {} ; - -/// Invalid type exception. -/** \ingroup exception - This exception will be thrown when the Object cannot convert its - underlying data to a specific type. The what() member function will - describe the expected and actual type of the data. -*/ -struct BadType : virtual Exception {} ; - -struct Unsupported : virtual Exception {} ; - -// Exception informations +/// Exception informations namespace expt { // back-trace information. should be present for all exceptions - typedef boost::error_info BacktraceInfo ; + typedef boost::error_info Backtrace_ ; - // generic error message - typedef boost::error_info ErrMsg ; - - // nested exception - typedef boost::error_info Nested ; + /// generic error message + typedef boost::error_info ErrMsg ; } } // end of namespace diff --git a/libgrive/src/util/StdioFile.cc b/libgrive/src/util/File.cc similarity index 63% rename from libgrive/src/util/StdioFile.cc rename to libgrive/src/util/File.cc index 435aea90..e6c047b3 100644 --- a/libgrive/src/util/StdioFile.cc +++ b/libgrive/src/util/File.cc @@ -17,27 +17,34 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "StdioFile.hh" +#include "File.hh" #include // boost headers #include #include -#include #include #include #include #include -#include #include #include #include +#ifdef WIN32 + #include + typedef int ssize_t ; +#else + #include +#endif + // local functions namespace { +using namespace gr ; + off_t LSeek( int fd, off_t offset, int whence ) { assert( fd >= 0 ) ; @@ -46,7 +53,7 @@ off_t LSeek( int fd, off_t offset, int whence ) if ( r == static_cast(-1) ) { BOOST_THROW_EXCEPTION( - gr::StdioFile::Error() + File::Error() << boost::errinfo_api_function("lseek") << boost::errinfo_errno(errno) ) ; @@ -61,7 +68,7 @@ struct stat FStat( int fd ) if ( ::fstat( fd, &s ) != 0 ) { BOOST_THROW_EXCEPTION( - gr::StdioFile::Error() + File::Error() << boost::errinfo_api_function("fstat") << boost::errinfo_errno(errno) ) ; @@ -71,28 +78,40 @@ struct stat FStat( int fd ) } // end of local functions -namespace gr { +namespace PROJ_NS { -StdioFile::StdioFile( ) : m_fd( -1 ) +File::File( ) : m_fd( -1 ) { } -StdioFile::StdioFile( const fs::path& path ) : m_fd( -1 ) +/** Opens the file for reading. + \param path Path to the file to be opened. + \throw Error When the file cannot be openned. +*/ +File::File( const fs::path& path ) : m_fd( -1 ) { OpenForRead( path ) ; } -StdioFile::StdioFile( const fs::path& path, int mode ) : m_fd( -1 ) +/** Opens the file for writing. + \param path Path to the file to be opened. + \param mode Mode of the file to be created, e.g. 0600 for user + readable/writable. + \throw Error When the file cannot be opened. +*/ +File::File( const fs::path& path, int mode ) : m_fd( -1 ) { OpenForWrite( path, mode ) ; } -StdioFile::~StdioFile( ) +/** The destructor will close the file. +*/ +File::~File( ) { Close() ; } -void StdioFile::Open( const fs::path& path, int flags, int mode ) +void File::Open( const fs::path& path, int flags, int mode ) { if ( IsOpened() ) Close() ; @@ -110,17 +129,25 @@ void StdioFile::Open( const fs::path& path, int flags, int mode ) } } -void StdioFile::OpenForRead( const fs::path& path ) +void File::OpenForRead( const fs::path& path ) { - Open( path, O_RDONLY, 0 ) ; + int flags = O_RDONLY ; +#ifdef WIN32 + flags |= O_BINARY ; +#endif + Open( path, flags, 0 ) ; } -void StdioFile::OpenForWrite( const fs::path& path, int mode ) +void File::OpenForWrite( const fs::path& path, int mode ) { - Open( path, O_CREAT|O_RDWR|O_TRUNC, mode ) ; + int flags = O_CREAT|O_RDWR|O_TRUNC ; +#ifdef WIN32 + flags |= O_BINARY ; +#endif + Open( path, flags, mode ) ; } -void StdioFile::Close() +void File::Close() { if ( IsOpened() ) { @@ -129,12 +156,15 @@ void StdioFile::Close() } } -bool StdioFile::IsOpened() const +bool File::IsOpened() const { return m_fd != -1 ; } -std::size_t StdioFile::Read( void *ptr, std::size_t size ) +/** Read bytes from file. See DataStream::Read() for details. + \throw Error In case of any error. +*/ +std::size_t File::Read( char *ptr, std::size_t size ) { assert( IsOpened() ) ; ssize_t count = ::read( m_fd, ptr, size ) ; @@ -149,7 +179,7 @@ std::size_t StdioFile::Read( void *ptr, std::size_t size ) return count ; } -std::size_t StdioFile::Write( const void *ptr, std::size_t size ) +std::size_t File::Write( const char *ptr, std::size_t size ) { assert( IsOpened() ) ; ssize_t count = ::write( m_fd, ptr, size ) ; @@ -157,26 +187,26 @@ std::size_t StdioFile::Write( const void *ptr, std::size_t size ) { BOOST_THROW_EXCEPTION( Error() - << boost::errinfo_api_function("read") + << boost::errinfo_api_function("write") << boost::errinfo_errno(errno) ) ; } return count ; } -off_t StdioFile::Seek( off_t offset, int whence ) +off_t File::Seek( off_t offset, int whence ) { assert( IsOpened() ) ; return LSeek( m_fd, offset, whence ) ; } -off_t StdioFile::Tell() const +off_t File::Tell() const { assert( IsOpened() ) ; return LSeek( m_fd, 0, SEEK_CUR ) ; } -u64_t StdioFile::Size() const +u64_t File::Size() const { assert( IsOpened() ) ; @@ -186,10 +216,10 @@ u64_t StdioFile::Size() const return static_cast( s.st_size ) ; } -void StdioFile::Chmod( int mode ) +void File::Chmod( int mode ) { assert( IsOpened() ) ; - +#ifndef WIN32 if ( ::fchmod( m_fd, mode ) != 0 ) { BOOST_THROW_EXCEPTION( @@ -198,12 +228,18 @@ void StdioFile::Chmod( int mode ) << boost::errinfo_errno(errno) ) ; } +#endif } -void* StdioFile::Map( off_t offset, std::size_t length ) +/// This function is not implemented in win32 yet. +void* File::Map( off_t offset, std::size_t length ) { assert( IsOpened() ) ; +#ifdef WIN32 + assert( false ) ; + return 0 ; +#else void *addr = ::mmap( 0, length, PROT_READ, MAP_PRIVATE, m_fd, offset ) ; if ( addr == reinterpret_cast( -1 ) ) { @@ -214,10 +250,12 @@ void* StdioFile::Map( off_t offset, std::size_t length ) ) ; } return addr ; +#endif } -void StdioFile::UnMap( void *addr, std::size_t length ) +void File::UnMap( void *addr, std::size_t length ) { +#ifndef WIN32 if ( ::munmap( addr, length ) != 0 ) { BOOST_THROW_EXCEPTION( @@ -226,6 +264,21 @@ void StdioFile::UnMap( void *addr, std::size_t length ) << boost::errinfo_errno(errno) ) ; } +#endif +} + +struct stat File::Stat() const +{ + struct stat result = {} ; + if ( ::fstat( m_fd, &result ) != 0 ) + { + BOOST_THROW_EXCEPTION( + Error() + << boost::errinfo_api_function("fstat") + << boost::errinfo_errno(errno) + ) ; + } + return result ; } } // end of namespace diff --git a/libgrive/src/util/StdioFile.hh b/libgrive/src/util/File.hh similarity index 66% rename from libgrive/src/util/StdioFile.hh rename to libgrive/src/util/File.hh index 21443b6f..041c5998 100644 --- a/libgrive/src/util/StdioFile.hh +++ b/libgrive/src/util/File.hh @@ -19,32 +19,47 @@ #pragma once +#include "DataStream.hh" #include "Exception.hh" #include "FileSystem.hh" #include "Types.hh" #include +struct stat ; + namespace gr { -class StdioFile +/** \brief A wrapper class for file read/write. + + It is a simple wrapper around the UNIX file descriptor. It will + throw exceptions (i.e. Error) when it encounters errors. +*/ +class File : public DataStream { public : + /// File specific errors. It often includes + /// boost::errinfo_api_function and boost::errinfo_errno for the + /// detail information. struct Error : virtual Exception {} ; public : - StdioFile() ; - StdioFile( const fs::path& path ) ; - StdioFile( const fs::path& path, int mode ) ; - ~StdioFile( ) ; + File() ; + File( const fs::path& path ) ; + File( const fs::path& path, int mode ) ; + ~File( ) ; + + File( const File& rhs ) ; + File& operator=( const File& rhs ) ; + void Swap( File& other ) ; void OpenForRead( const fs::path& path ) ; void OpenForWrite( const fs::path& path, int mode = 0600 ) ; void Close() ; bool IsOpened() const ; - std::size_t Read( void *ptr, std::size_t size ) ; - std::size_t Write( const void *ptr, std::size_t size ) ; + std::size_t Read( char *ptr, std::size_t size ) ; + std::size_t Write( const char *ptr, std::size_t size ) ; off_t Seek( off_t offset, int whence ) ; off_t Tell() const ; @@ -54,7 +69,9 @@ public : void* Map( off_t offset, std::size_t length ) ; static void UnMap( void *addr, std::size_t length ) ; - + + struct stat Stat() const ; + private : void Open( const fs::path& path, int flags, int mode ) ; diff --git a/libgrive/src/util/FileSystem.hh b/libgrive/src/util/FileSystem.hh index f6c18acd..9eb02d3b 100644 --- a/libgrive/src/util/FileSystem.hh +++ b/libgrive/src/util/FileSystem.hh @@ -25,17 +25,4 @@ namespace gr { namespace fs = boost::filesystem ; - - // these two functions are for ancient distro which does not have boost v1.44 or later - // will be removed once people upgrade - - inline std::string Path2Str( const fs::path& p ) - { - return p.string() ; - } - - inline std::string Path2Str( const std::string& s ) - { - return s ; - } } diff --git a/libgrive/src/util/MemMap.cc b/libgrive/src/util/MemMap.cc index 6f8b4c77..12b1f780 100644 --- a/libgrive/src/util/MemMap.cc +++ b/libgrive/src/util/MemMap.cc @@ -18,11 +18,11 @@ */ #include "MemMap.hh" -#include "StdioFile.hh" +#include "File.hh" namespace gr { -MemMap::MemMap( StdioFile& file, off_t offset, std::size_t length ) : +MemMap::MemMap( File& file, off_t offset, std::size_t length ) : m_addr ( file.Map( offset, length ) ), m_length( length ) { @@ -30,7 +30,7 @@ MemMap::MemMap( StdioFile& file, off_t offset, std::size_t length ) : MemMap::~MemMap() { - StdioFile::UnMap( m_addr, m_length ) ; + File::UnMap( m_addr, m_length ) ; } void* MemMap::Addr() const diff --git a/libgrive/src/util/MemMap.hh b/libgrive/src/util/MemMap.hh index 91a30335..b8ca5a26 100644 --- a/libgrive/src/util/MemMap.hh +++ b/libgrive/src/util/MemMap.hh @@ -25,7 +25,7 @@ namespace gr { -class StdioFile ; +class File ; class MemMap { @@ -33,7 +33,7 @@ public : struct Error : virtual Exception {} ; public : - MemMap( StdioFile& file, off_t offset, std::size_t length ) ; + MemMap( File& file, off_t offset, std::size_t length ) ; ~MemMap() ; void* Addr() const ; From 39b2b4fb0ba9f3551702f18e7b33830f0740d92d Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Sun, 28 Apr 2013 19:40:06 +0800 Subject: [PATCH 033/166] added DriveModel. just skeleton. --- bgrive/src/DriveModel.cc | 71 ++++++++++++++++++++++++++++++++++++++++ bgrive/src/DriveModel.hh | 43 ++++++++++++++++++++++++ bgrive/src/MainWnd.cc | 4 +++ bgrive/src/MainWnd.hh | 5 ++- bgrive/src/main.cc | 4 +-- 5 files changed, 124 insertions(+), 3 deletions(-) create mode 100644 bgrive/src/DriveModel.cc create mode 100644 bgrive/src/DriveModel.hh diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc new file mode 100644 index 00000000..e207fb4f --- /dev/null +++ b/bgrive/src/DriveModel.cc @@ -0,0 +1,71 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "DriveModel.hh" + +#include + +namespace gr { + +DriveModel::DriveModel( ) +{ +} + +Qt::ItemFlags DriveModel::flags( const QModelIndex& ) const +{ + return Qt::ItemIsEnabled | Qt::ItemIsSelectable ; +} + +QVariant DriveModel::data( const QModelIndex& index, int role ) const +{ + return role == Qt::DisplayRole ? QString("wow") : QVariant() ; +} + +QVariant DriveModel::headerData( int section, Qt::Orientation orientation, int role ) const +{ + return role == Qt::DisplayRole ? QString("header") : QVariant() ; +} + +int DriveModel::rowCount( const QModelIndex& parent ) const +{ + return 10 ; +} + +int DriveModel::columnCount( const QModelIndex& parent ) const +{ + return 1 ; +} + +bool DriveModel::hasChildren( const QModelIndex& parent ) const +{ + return parent.isValid() ? false : true ; +} + +QModelIndex DriveModel::index( int row, int column, const QModelIndex & parent ) const +{ + return parent.isValid() ? QModelIndex() : createIndex( row, column, 0 ) ; +} + +QModelIndex DriveModel::parent( const QModelIndex& idx ) const +{ + return QModelIndex() ; +} + +} // end of namespace diff --git a/bgrive/src/DriveModel.hh b/bgrive/src/DriveModel.hh new file mode 100644 index 00000000..cb5fd208 --- /dev/null +++ b/bgrive/src/DriveModel.hh @@ -0,0 +1,43 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include + +namespace gr { + +class DriveModel : public QAbstractItemModel +{ +public : + DriveModel( ) ; + + Qt::ItemFlags flags( const QModelIndex & index ) const ; + QVariant data( const QModelIndex& index, int role ) const ; + QVariant headerData( int section, Qt::Orientation orientation, int role ) const ; + int rowCount( const QModelIndex& parent ) const ; + int columnCount( const QModelIndex& parent ) const ; + bool hasChildren ( const QModelIndex& parent ) const ; + QModelIndex index( int row, int column, const QModelIndex& parent ) const ; + QModelIndex parent( const QModelIndex& idx ) const ; +} ; + +} // end of namespace + diff --git a/bgrive/src/MainWnd.cc b/bgrive/src/MainWnd.cc index 1712ebc8..d73c9c95 100644 --- a/bgrive/src/MainWnd.cc +++ b/bgrive/src/MainWnd.cc @@ -20,11 +20,15 @@ #include "MainWnd.hh" +#include + namespace gr { MainWnd::MainWnd( ) { m_ui.setupUi(this) ; + + m_ui.m_dir->setModel( &m_drive ) ; } } // end of namespace diff --git a/bgrive/src/MainWnd.hh b/bgrive/src/MainWnd.hh index f50d431e..d5be6644 100644 --- a/bgrive/src/MainWnd.hh +++ b/bgrive/src/MainWnd.hh @@ -23,6 +23,8 @@ #include #include "ui_MainWindow.h" +#include "DriveModel.hh" + namespace gr { class MainWnd : public QMainWindow @@ -33,7 +35,8 @@ public : MainWnd( ) ; private : - Ui::MainWindow m_ui ; + Ui::MainWindow m_ui ; + DriveModel m_drive ; } ; } // end of namespace diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc index 4b61317b..9b85d159 100644 --- a/bgrive/src/main.cc +++ b/bgrive/src/main.cc @@ -55,7 +55,7 @@ int main( int argc, char **argv ) OAuth2 token( refresh_token, client_id, client_secret ) ; AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; - +/* http::XmlResponse xml ; agent.Get( feed_base + "/-/folder?max-results=50&showroot=true", &xml, http::Header() ) ; @@ -69,7 +69,7 @@ int main( int argc, char **argv ) qDebug() << e.Name().c_str() ; } } while ( feed.GetNext( &agent, http::Header() ) ) ; - +*/ QApplication app( argc, argv ) ; MainWnd wnd ; wnd.show(); From d4a7fae2735037e8acee9b779db6b5d2cacae963 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Sun, 28 Apr 2013 23:50:20 +0800 Subject: [PATCH 034/166] refactored exception handling --- CMakeLists.txt | 1 - grive/src/main.cc | 2 +- libgrive/src/http/CurlAgent.cc | 4 ++-- libgrive/src/http/Error.hh | 7 +++++-- libgrive/src/protocol/Json.cc | 2 +- libgrive/src/protocol/Json.hh | 1 + libgrive/src/util/Config.cc | 2 +- libgrive/src/util/Config.hh | 2 +- libgrive/src/util/Crypt.cc | 5 ++++- libgrive/src/util/Crypt.hh | 6 ++++++ libgrive/src/util/Exception.cc | 2 +- libgrive/src/util/Exception.hh | 5 +---- libgrive/src/util/File.cc | 2 +- libgrive/src/xml/Node.cc | 2 +- libgrive/src/xml/Node.hh | 4 ++++ libgrive/src/xml/NodeSet.cc | 2 +- libgrive/src/xml/NodeSet.hh | 4 ++++ libgrive/src/xml/TreeBuilder.cc | 4 ++-- libgrive/src/xml/TreeBuilder.hh | 6 ++++++ 19 files changed, 43 insertions(+), 20 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 7d16f2de..48f1460b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -6,7 +6,6 @@ set( GRIVE_VERSION "0.3.0-pre" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) add_definitions( -D_FILE_OFFSET_BITS=64 ) -add_definitions( -DPROJ_NS=gr ) add_subdirectory( libgrive ) add_subdirectory( grive ) diff --git a/grive/src/main.cc b/grive/src/main.cc index cf3a3d1a..b2c43196 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -54,7 +54,7 @@ namespace po = boost::program_options; void InitGCrypt() { if ( !gcry_check_version(GCRYPT_VERSION) ) - throw Exception() << expt::ErrMsg( "libgcrypt version mismatch" ) ; + throw std::runtime_error( "libgcrypt version mismatch" ) ; // disable secure memory gcry_control(GCRYCTL_DISABLE_SECMEM, 0); diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 2cf78b81..87b49c17 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -150,7 +150,7 @@ long CurlAgent::ExecCurl( dest->Clear() ; CURLcode curl_code = ::curl_easy_perform(curl); - // get the HTTTP response code + // get the HTTP response code long http_code = 0; ::curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code); Trace( "HTTP response %1%", http_code ) ; @@ -165,7 +165,7 @@ long CurlAgent::ExecCurl( Error() << CurlCode( curl_code ) << Url( url ) - << expt::ErrMsg( error ) + << CurlErrMsg( error ) << HttpHeader( hdr ) ) ; } diff --git a/libgrive/src/http/Error.hh b/libgrive/src/http/Error.hh index 9b41376e..815ee04f 100644 --- a/libgrive/src/http/Error.hh +++ b/libgrive/src/http/Error.hh @@ -27,13 +27,16 @@ namespace gr { namespace http { struct Error : virtual Exception {} ; // CURL error code -typedef boost::error_info CurlCode ; +typedef boost::error_info CurlCode ; + +// CURL error message +typedef boost::error_info CurlErrMsg ; // URL typedef boost::error_info Url ; // HTTP headers -typedef boost::error_info HttpHeader ; +typedef boost::error_info HttpHeader ; // HTTP response code typedef boost::error_info HttpResponse ; diff --git a/libgrive/src/protocol/Json.cc b/libgrive/src/protocol/Json.cc index 58da219b..d290924c 100644 --- a/libgrive/src/protocol/Json.cc +++ b/libgrive/src/protocol/Json.cc @@ -515,7 +515,7 @@ Json Json::Parse( DataStream *in ) BOOST_THROW_EXCEPTION( Error() << JsonCApi_( "json_tokener_parse" ) - << expt::ErrMsg( ::json_tokener_error_desc(err) ) + << ErrMsg_( ::json_tokener_error_desc(err) ) ) ; } diff --git a/libgrive/src/protocol/Json.hh b/libgrive/src/protocol/Json.hh index a1f6bf0e..1ebf1737 100644 --- a/libgrive/src/protocol/Json.hh +++ b/libgrive/src/protocol/Json.hh @@ -48,6 +48,7 @@ public : typedef boost::error_info KeyNotFound_ ; typedef boost::error_info JsonCApi_ ; typedef boost::error_info Value_ ; + typedef boost::error_info ErrMsg_ ; template struct Val_ diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index 6111b1d4..b5a9f15b 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -28,7 +28,7 @@ namespace po = boost::program_options; -namespace PROJ_NS { +namespace gr { const std::string default_filename = ".grive"; const char *env_name = "GR_CONFIG"; diff --git a/libgrive/src/util/Config.hh b/libgrive/src/util/Config.hh index 6057f148..e2c0423f 100644 --- a/libgrive/src/util/Config.hh +++ b/libgrive/src/util/Config.hh @@ -31,7 +31,7 @@ namespace boost } } -namespace PROJ_NS { +namespace gr { class Config { diff --git a/libgrive/src/util/Crypt.cc b/libgrive/src/util/Crypt.cc index f0049f57..5fb8ebe8 100644 --- a/libgrive/src/util/Crypt.cc +++ b/libgrive/src/util/Crypt.cc @@ -45,7 +45,10 @@ MD5::MD5() : m_impl( new Impl ) ::gcry_error_t err = ::gcry_md_open( &m_impl->hd, GCRY_MD_MD5, 0 ) ; if ( err != GPG_ERR_NO_ERROR ) { - BOOST_THROW_EXCEPTION( Exception() << expt::ErrMsg( ::gcry_strerror(err) ) ) ; + BOOST_THROW_EXCEPTION( Exception() + << GCryptErr_( ::gcry_strerror(err) ) + << GCryptApi_( "gcry_md_open" ) + ) ; } } diff --git a/libgrive/src/util/Crypt.hh b/libgrive/src/util/Crypt.hh index 6cd6ceba..dcd8dd34 100644 --- a/libgrive/src/util/Crypt.hh +++ b/libgrive/src/util/Crypt.hh @@ -19,6 +19,8 @@ #pragma once +#include "util/Exception.hh" + #include #include @@ -32,6 +34,10 @@ namespace crypt { class MD5 { +public : + typedef boost::error_info GCryptErr_ ; + typedef boost::error_info GCryptApi_ ; + public : MD5() ; ~MD5() ; diff --git a/libgrive/src/util/Exception.cc b/libgrive/src/util/Exception.cc index a410bbb5..9cf66810 100644 --- a/libgrive/src/util/Exception.cc +++ b/libgrive/src/util/Exception.cc @@ -28,7 +28,7 @@ #include #include -namespace PROJ_NS { +namespace gr { class Backtrace ; diff --git a/libgrive/src/util/Exception.hh b/libgrive/src/util/Exception.hh index 164f9dd5..f1515841 100644 --- a/libgrive/src/util/Exception.hh +++ b/libgrive/src/util/Exception.hh @@ -25,7 +25,7 @@ #include #include -namespace PROJ_NS { +namespace gr { class Backtrace ; @@ -51,9 +51,6 @@ namespace expt { // back-trace information. should be present for all exceptions typedef boost::error_info Backtrace_ ; - - /// generic error message - typedef boost::error_info ErrMsg ; } } // end of namespace diff --git a/libgrive/src/util/File.cc b/libgrive/src/util/File.cc index e6c047b3..375cd625 100644 --- a/libgrive/src/util/File.cc +++ b/libgrive/src/util/File.cc @@ -78,7 +78,7 @@ struct stat FStat( int fd ) } // end of local functions -namespace PROJ_NS { +namespace gr { File::File( ) : m_fd( -1 ) { diff --git a/libgrive/src/xml/Node.cc b/libgrive/src/xml/Node.cc index be12d312..abed0c4e 100644 --- a/libgrive/src/xml/Node.cc +++ b/libgrive/src/xml/Node.cc @@ -92,7 +92,7 @@ public : // cannot allow duplicate attribute nodes if ( child->m_type == attr && p.first != p.second ) - BOOST_THROW_EXCEPTION( Error() << expt::ErrMsg( "duplicate attribute " + child->m_name ) ) ; + BOOST_THROW_EXCEPTION( Error() << DupAttr_( child->m_name ) ) ; vec.insert( p.second, child ) ; } diff --git a/libgrive/src/xml/Node.hh b/libgrive/src/xml/Node.hh index 85c2a6f8..fb4b6c43 100644 --- a/libgrive/src/xml/Node.hh +++ b/libgrive/src/xml/Node.hh @@ -19,6 +19,8 @@ #pragma once +#include "util/Exception.hh" + #include #include @@ -39,6 +41,8 @@ private : public : class iterator ; + typedef boost::error_info DupAttr_ ; + public : Node() ; Node( const Node& node ) ; diff --git a/libgrive/src/xml/NodeSet.cc b/libgrive/src/xml/NodeSet.cc index 8ccc82fd..78b74931 100644 --- a/libgrive/src/xml/NodeSet.cc +++ b/libgrive/src/xml/NodeSet.cc @@ -124,7 +124,7 @@ NodeSet NodeSet::operator[]( const std::string& name ) const Node NodeSet::front() const { if ( empty() ) - throw Error() << expt::ErrMsg( "empty node set" ) ; + BOOST_THROW_EXCEPTION( Error() << EmptyNodeSet_(0) ) ; return *m_first ; } diff --git a/libgrive/src/xml/NodeSet.hh b/libgrive/src/xml/NodeSet.hh index 98cb97c9..8820a8fd 100644 --- a/libgrive/src/xml/NodeSet.hh +++ b/libgrive/src/xml/NodeSet.hh @@ -19,6 +19,8 @@ #pragma once +#include "util/Exception.hh" + #include "Node.hh" #include @@ -32,6 +34,8 @@ class NodeSet public : typedef Node::iterator iterator ; + typedef boost::error_info EmptyNodeSet_ ; + public : NodeSet() ; NodeSet( const NodeSet& n ) ; diff --git a/libgrive/src/xml/TreeBuilder.cc b/libgrive/src/xml/TreeBuilder.cc index 66be3344..3a31b65b 100644 --- a/libgrive/src/xml/TreeBuilder.cc +++ b/libgrive/src/xml/TreeBuilder.cc @@ -69,7 +69,7 @@ Node TreeBuilder::ParseFile( const std::string& file ) void TreeBuilder::ParseData( const char *data, std::size_t count, bool last ) { if ( ::XML_Parse( m_impl->psr, data, count, last ) == 0 ) - throw Error() << expt::ErrMsg( "XML parse error" ) ; + BOOST_THROW_EXCEPTION( Error() << ExpatApiError("XML_Parse") ); } Node TreeBuilder::Parse( const std::string& xml ) @@ -85,7 +85,7 @@ Node TreeBuilder::Result() const assert( m_impl->stack.size() == 1 ) ; if ( m_impl->stack.front().size() != 1 ) - throw Error() << expt::ErrMsg( "invalid node" ) ; + BOOST_THROW_EXCEPTION( Error() << LogicError(0) ) ; return *m_impl->stack.front().begin() ; } diff --git a/libgrive/src/xml/TreeBuilder.hh b/libgrive/src/xml/TreeBuilder.hh index 184b55f6..c2a04f10 100644 --- a/libgrive/src/xml/TreeBuilder.hh +++ b/libgrive/src/xml/TreeBuilder.hh @@ -19,6 +19,8 @@ #pragma once +#include "util/Exception.hh" + #include #include @@ -28,6 +30,10 @@ class Node ; class TreeBuilder { +public : + typedef boost::error_info ExpatApiError ; + typedef boost::error_info LogicError ; + public : TreeBuilder() ; ~TreeBuilder() ; From 443a85d558c33b62a8af8d4ffb033584a0e320fa Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 00:13:34 +0800 Subject: [PATCH 035/166] removing redundant header parameter in Feed --- bgrive/src/DriveModel.hh | 1 + bgrive/src/main.cc | 12 +++++------- libgrive/src/drive/Drive.cc | 8 ++++---- libgrive/src/drive/Feed.cc | 9 +++++---- libgrive/src/drive/Feed.hh | 6 +++--- 5 files changed, 18 insertions(+), 18 deletions(-) diff --git a/bgrive/src/DriveModel.hh b/bgrive/src/DriveModel.hh index cb5fd208..f7ee26ff 100644 --- a/bgrive/src/DriveModel.hh +++ b/bgrive/src/DriveModel.hh @@ -29,6 +29,7 @@ class DriveModel : public QAbstractItemModel public : DriveModel( ) ; + // QAbstractItemModel overrides Qt::ItemFlags flags( const QModelIndex & index ) const ; QVariant data( const QModelIndex& index, int role ) const ; QVariant headerData( int section, Qt::Orientation orientation, int role ) const ; diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc index 9b85d159..0c945f36 100644 --- a/bgrive/src/main.cc +++ b/bgrive/src/main.cc @@ -29,7 +29,7 @@ #include "http/CurlAgent.hh" #include "http/Header.hh" -#include "http/XmlResponse.hh" +//#include "http/XmlResponse.hh" #include "protocol/Json.hh" #include "protocol/OAuth2.hh" @@ -55,11 +55,9 @@ int main( int argc, char **argv ) OAuth2 token( refresh_token, client_id, client_secret ) ; AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; -/* - http::XmlResponse xml ; - agent.Get( feed_base + "/-/folder?max-results=50&showroot=true", &xml, http::Header() ) ; + Feed feed ; + feed.Start( &agent, feed_base + "/-/folder?max-results=50&showroot=true" ) ; - Feed feed( xml.Response() ) ; do { // first, get all collections from the query result @@ -68,8 +66,8 @@ int main( int argc, char **argv ) Entry e( *i ) ; qDebug() << e.Name().c_str() ; } - } while ( feed.GetNext( &agent, http::Header() ) ) ; -*/ + } while ( feed.GetNext( &agent ) ) ; + QApplication app( argc, argv ) ; MainWnd wnd ; wnd.show(); diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index 82cfc749..4fa2cbf0 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -120,7 +120,7 @@ void Drive::SyncFolders( ) m_state.FromRemote( e ) ; } } - } while ( feed.GetNext( m_http, http::Header() ) ) ; + } while ( feed.GetNext( m_http ) ) ; m_state.ResolveEntry() ; } @@ -140,7 +140,7 @@ void Drive::DetectChanges() if ( m_options["log-xml"].Bool() ) feed.EnableLog( "/tmp/file", ".xml" ) ; - feed.Start( m_http, http::Header(), feed_base + "?showfolders=true&showroot=true" ) ; + feed.Start( m_http, feed_base + "?showfolders=true&showroot=true" ) ; m_resume_link = feed.Root()["link"]. Find( "@rel", "http://schemas.google.com/g/2005#resumable-create-media" )["@href"] ; @@ -151,7 +151,7 @@ void Drive::DetectChanges() feed.begin(), feed.end(), boost::bind( &Drive::FromRemote, this, _1 ) ) ; - } while ( feed.GetNext( m_http, http::Header() ) ) ; + } while ( feed.GetNext( m_http ) ) ; // pull the changes feed if ( prev_stamp != -1 ) @@ -161,7 +161,7 @@ void Drive::DetectChanges() if ( m_options["log-xml"].Bool() ) feed.EnableLog( "/tmp/changes", ".xml" ) ; - feed.Start( m_http, http::Header(), ChangesFeed(prev_stamp+1) ) ; + feed.Start( m_http, ChangesFeed(prev_stamp+1) ) ; std::for_each( changes.begin(), changes.end(), diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed.cc index a0fe9568..9c74bacc 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed.cc @@ -20,6 +20,7 @@ #include "Feed.hh" #include "http/Agent.hh" +#include "http/Header.hh" #include "http/ResponseLog.hh" #include "http/XmlResponse.hh" #include "xml/NodeSet.hh" @@ -62,7 +63,7 @@ std::string Feed::Next() const return nss.empty() ? "" : std::string(nss["@href"]) ; } -void Feed::Start( http::Agent *http, const http::Header& auth, const std::string& url ) +void Feed::Start( http::Agent *http, const std::string& url ) { http::XmlResponse xrsp ; http::ResponseLog log( &xrsp ) ; @@ -73,20 +74,20 @@ void Feed::Start( http::Agent *http, const http::Header& auth, const std::string (boost::format( "-#%1%%2%" ) % m_log->sequence++ % m_log->suffix ).str(), &xrsp ) ; - http->Get( url, &log, auth ) ; + http->Get( url, &log, http::Header() ) ; m_root = xrsp.Response() ; m_entries = m_root["entry"] ; } -bool Feed::GetNext( http::Agent *http, const http::Header& auth ) +bool Feed::GetNext( http::Agent *http ) { assert( http != 0 ) ; xml::NodeSet nss = m_root["link"].Find( "@rel", "next" ) ; if ( !nss.empty() ) { - Start( http, auth, nss["@href"] ) ; + Start( http, nss["@href"] ) ; return true ; } else diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/drive/Feed.hh index deaf18c1..ba98a37f 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/drive/Feed.hh @@ -44,7 +44,7 @@ public : public : explicit Feed( const xml::Node& root ) ; Feed( ) ; - void Start( http::Agent *http, const http::Header& auth, const std::string& url ) ; + void Start( http::Agent *http, const std::string& url ) ; void Assign( const xml::Node& root ) ; const xml::Node& Root() const ; @@ -53,7 +53,7 @@ public : iterator end() const ; std::string Next() const ; - bool GetNext( http::Agent *http, const http::Header& auth ) ; + bool GetNext( http::Agent *http ) ; void EnableLog( const std::string& prefix, const std::string& suffix ) ; @@ -67,7 +67,7 @@ private : std::auto_ptr m_log ; xml::Node m_root ; - xml::NodeSet m_entries ; + xml::NodeSet m_entries ; } ; class Feed::iterator : public boost::iterator_adaptor< From 738435837be79feff43e6674672fc91185fad049 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 01:25:34 +0800 Subject: [PATCH 036/166] using DataStream instead of Receiveable --- libgrive/src/drive/Resource.cc | 2 +- libgrive/src/http/Agent.hh | 14 +++++----- libgrive/src/http/CurlAgent.cc | 39 ++++++++++++++------------- libgrive/src/http/CurlAgent.hh | 22 ++++++++------- libgrive/src/http/Download.cc | 10 +++++-- libgrive/src/http/Download.hh | 6 ++--- libgrive/src/http/MarshalAgent.hh | 39 --------------------------- libgrive/src/http/Receivable.hh | 34 ----------------------- libgrive/src/http/ResponseLog.cc | 16 +++++------ libgrive/src/http/ResponseLog.hh | 18 ++++++------- libgrive/src/http/StringResponse.cc | 9 +++++-- libgrive/src/http/StringResponse.hh | 7 ++--- libgrive/src/http/XmlResponse.cc | 9 ++++--- libgrive/src/http/XmlResponse.hh | 7 ++--- libgrive/src/protocol/AuthAgent.cc | 12 ++++----- libgrive/src/protocol/AuthAgent.hh | 12 ++++----- libgrive/src/protocol/JsonResponse.cc | 8 +++--- libgrive/src/protocol/JsonResponse.hh | 8 +++--- libgrive/src/xml/NodeSet.cc | 4 +++ libgrive/src/xml/TreeBuilder.cc | 4 +++ libgrive/src/xml/TreeBuilder.hh | 2 ++ 21 files changed, 118 insertions(+), 164 deletions(-) delete mode 100644 libgrive/src/http/MarshalAgent.hh delete mode 100644 libgrive/src/http/Receivable.hh diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 63a54bd6..b4220593 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -613,7 +613,7 @@ bool Resource::Upload( std::string uplink = http->RedirLocation() ; http::XmlResponse xml ; - http->Put( uplink, file, &xml, uphdr ) ; + http->Put( uplink, &file, &xml, uphdr ) ; AssignIDs( Entry( xml.Response() ) ) ; m_mtime = Entry(xml.Response()).MTime(); diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index 0ca1e9b7..a1903ce9 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -23,12 +23,12 @@ namespace gr { +class DataStream ; class File ; namespace http { class Header ; -class Receivable ; class Agent { @@ -38,30 +38,30 @@ public : virtual long Put( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) = 0 ; virtual long Put( const std::string& url, - File& file, - Receivable *dest, + File *file, + DataStream *dest, const Header& hdr ) = 0 ; virtual long Get( const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) = 0 ; virtual long Post( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) = 0 ; virtual long Custom( const std::string& method, const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) = 0 ; virtual std::string RedirLocation() const = 0 ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 87b49c17..d725e86c 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -19,12 +19,11 @@ #include "CurlAgent.hh" -#include "Download.hh" #include "Error.hh" #include "Header.hh" -#include "Receivable.hh" #include "util/log/Log.hh" +#include "util/DataStream.hh" #include "util/File.hh" #include @@ -47,7 +46,7 @@ namespace { using namespace gr::http ; using namespace gr ; -size_t ReadStringCallback( void *ptr, std::size_t size, std::size_t nmemb, std::string *data ) +std::size_t ReadStringCallback( void *ptr, std::size_t size, std::size_t nmemb, std::string *data ) { assert( ptr != 0 ) ; assert( data != 0 ) ; @@ -62,14 +61,14 @@ size_t ReadStringCallback( void *ptr, std::size_t size, std::size_t nmemb, std:: return count ; } -size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, File *file ) +std::size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, File *file ) { assert( ptr != 0 ) ; assert( file != 0 ) ; - u64_t count = std::min( - static_cast(size * nmemb), - static_cast(file->Size() - file->Tell()) ) ; + std::size_t count = std::min( + static_cast(size * nmemb), + static_cast(file->Size() - file->Tell()) ) ; assert( count <= std::numeric_limits::max() ) ; if ( count > 0 ) @@ -125,15 +124,15 @@ std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, Cur return size*nmemb ; } -std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, Receivable *recv ) +std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, DataStream *recv ) { assert( recv != 0 ) ; - return recv->OnData( ptr, size * nmemb ) ; + return recv->Write( static_cast(ptr), size * nmemb ) ; } long CurlAgent::ExecCurl( const std::string& url, - Receivable *dest, + DataStream *dest, const http::Header& hdr ) { CURL *curl = m_pimpl->curl ; @@ -147,7 +146,7 @@ long CurlAgent::ExecCurl( SetHeader( hdr ) ; - dest->Clear() ; +// dest->Clear() ; CURLcode curl_code = ::curl_easy_perform(curl); // get the HTTP response code @@ -176,7 +175,7 @@ long CurlAgent::ExecCurl( long CurlAgent::Put( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Trace("HTTP PUT \"%1%\"", url ) ; @@ -197,10 +196,12 @@ long CurlAgent::Put( long CurlAgent::Put( const std::string& url, - File& file, - Receivable *dest, + File *file, + DataStream *dest, const Header& hdr ) { + assert( file != 0 ) ; + Trace("HTTP PUT \"%1%\"", url ) ; Init() ; @@ -209,15 +210,15 @@ long CurlAgent::Put( // set common options ::curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L ) ; ::curl_easy_setopt(curl, CURLOPT_READFUNCTION, &ReadFileCallback ) ; - ::curl_easy_setopt(curl, CURLOPT_READDATA , &file ) ; - ::curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, static_cast(file.Size()) ) ; + ::curl_easy_setopt(curl, CURLOPT_READDATA , file ) ; + ::curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, static_cast(file->Size()) ) ; return ExecCurl( url, dest, hdr ) ; } long CurlAgent::Get( const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Trace("HTTP GET \"%1%\"", url ) ; @@ -232,7 +233,7 @@ long CurlAgent::Get( long CurlAgent::Post( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Trace("HTTP POST \"%1%\" with \"%2%\"", url, data ) ; @@ -254,7 +255,7 @@ long CurlAgent::Post( long CurlAgent::Custom( const std::string& method, const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Trace("HTTP %2% \"%1%\"", url, method ) ; diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index 2b9a394a..c142c3b1 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -24,9 +24,11 @@ #include #include -namespace gr { namespace http { +namespace gr { -class Receivable ; +class DataStream ; + +namespace http { /*! \brief agent to provide HTTP access @@ -42,30 +44,30 @@ public : long Put( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) ; long Put( const std::string& url, - File& file, - Receivable *dest, + File *file, + DataStream *dest, const Header& hdr ) ; long Get( const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) ; long Post( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) ; long Custom( const std::string& method, const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) ; std::string RedirLocation() const ; @@ -75,12 +77,12 @@ public : private : static std::size_t HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; - static std::size_t Receive( void* ptr, size_t size, size_t nmemb, Receivable *recv ) ; + static std::size_t Receive( void* ptr, size_t size, size_t nmemb, DataStream *recv ) ; void SetHeader( const Header& hdr ) ; long ExecCurl( const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) ; void Init() ; diff --git a/libgrive/src/http/Download.cc b/libgrive/src/http/Download.cc index 3b98ac4d..885187ef 100644 --- a/libgrive/src/http/Download.cc +++ b/libgrive/src/http/Download.cc @@ -65,14 +65,20 @@ std::string Download::Finish() const return m_crypt.get() != 0 ? m_crypt->Get() : "" ; } -std::size_t Download::OnData( void *data, std::size_t count ) +std::size_t Download::Write( const char *data, std::size_t count ) { assert( data != 0 ) ; if ( m_crypt.get() != 0 ) m_crypt->Write( data, count ) ; - return m_file.Write( static_cast(data), count ) ; + return m_file.Write( data, count ) ; +} + + +std::size_t Download::Read( char *data, std::size_t count ) +{ + return count ; } } } // end of namespace diff --git a/libgrive/src/http/Download.hh b/libgrive/src/http/Download.hh index 683f864a..77cc1ae8 100644 --- a/libgrive/src/http/Download.hh +++ b/libgrive/src/http/Download.hh @@ -19,7 +19,6 @@ #pragma once -#include "Receivable.hh" #include "util/File.hh" #include @@ -33,7 +32,7 @@ namespace crypt namespace http { -class Download : public http::Receivable +class Download : public DataStream { public : struct NoChecksum {} ; @@ -44,7 +43,8 @@ public : std::string Finish() const ; void Clear() ; - std::size_t OnData( void *data, std::size_t count ) ; + std::size_t Write( const char *data, std::size_t count ) ; + std::size_t Read( char *, std::size_t ) ; private : File m_file ; diff --git a/libgrive/src/http/MarshalAgent.hh b/libgrive/src/http/MarshalAgent.hh deleted file mode 100644 index 0ca2b272..00000000 --- a/libgrive/src/http/MarshalAgent.hh +++ /dev/null @@ -1,39 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include "http/Agent.hh" - -namespace gr { namespace http { - -class HttpRequest ; - -/*! \brief An marshaller for HTTP agent - - This agent will marshal the HTTP requests to a request objects containing the arguments - of the request and the response. -*/ -class MarshalAgent : public Agent -{ -public : - MarshalAgent( std::auto_ptr real_agent ) ; -} ; - -}} // end of namespace diff --git a/libgrive/src/http/Receivable.hh b/libgrive/src/http/Receivable.hh deleted file mode 100644 index 0eb1afb5..00000000 --- a/libgrive/src/http/Receivable.hh +++ /dev/null @@ -1,34 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include - -namespace gr { namespace http { - -class Receivable -{ -public : - virtual ~Receivable() {} - virtual std::size_t OnData( void *data, std::size_t count ) = 0 ; - virtual void Clear() = 0 ; -} ; - -} } // end of namespace diff --git a/libgrive/src/http/ResponseLog.cc b/libgrive/src/http/ResponseLog.cc index 3384bf7e..9de4ac04 100644 --- a/libgrive/src/http/ResponseLog.cc +++ b/libgrive/src/http/ResponseLog.cc @@ -29,35 +29,35 @@ namespace gr { namespace http { ResponseLog::ResponseLog( const std::string& prefix, const std::string& suffix, - Receivable *next ) : + DataStream *next ) : m_enabled ( true ), m_next ( next ) { Reset( prefix, suffix, next ) ; } -ResponseLog::ResponseLog( Receivable *next ) : +ResponseLog::ResponseLog( DataStream *next ) : m_enabled ( false ), m_next ( next ) { assert( m_next != 0 ) ; } -std::size_t ResponseLog::OnData( void *data, std::size_t count ) +std::size_t ResponseLog::Write( const char *data, std::size_t count ) { if ( m_enabled ) { assert( m_log.rdbuf() != 0 ) ; - m_log.rdbuf()->sputn( reinterpret_cast(data), count ) ; + m_log.rdbuf()->sputn( data, count ) ; } - return m_next->OnData( data, count ) ; + return m_next->Write( data, count ) ; } -void ResponseLog::Clear() +std::size_t ResponseLog::Read( char *data, std::size_t count ) { assert( m_next != 0 ) ; - m_next->Clear() ; + return m_next->Read( data, count ) ; } std::string ResponseLog::Filename( const std::string& prefix, const std::string& suffix ) @@ -70,7 +70,7 @@ void ResponseLog::Enable( bool enable ) m_enabled = enable ; } -void ResponseLog::Reset( const std::string& prefix, const std::string& suffix, Receivable *next ) +void ResponseLog::Reset( const std::string& prefix, const std::string& suffix, DataStream *next ) { assert( next != 0 ) ; diff --git a/libgrive/src/http/ResponseLog.hh b/libgrive/src/http/ResponseLog.hh index 9b7de656..046b5c97 100644 --- a/libgrive/src/http/ResponseLog.hh +++ b/libgrive/src/http/ResponseLog.hh @@ -19,27 +19,27 @@ #pragma once -#include "Receivable.hh" +#include "util/DataStream.hh" #include #include namespace gr { namespace http { -class ResponseLog : public Receivable +class ResponseLog : public DataStream { public : ResponseLog( const std::string& prefix, const std::string& suffix, - Receivable *next ) ; - ResponseLog( Receivable *next ) ; + DataStream *next ) ; + ResponseLog( DataStream *next ) ; - std::size_t OnData( void *data, std::size_t count ) ; - void Clear() ; - + std::size_t Write( const char *data, std::size_t count ) ; + std::size_t Read( char *data, std::size_t count ) ; + void Enable( bool enable = true ) ; - void Reset( const std::string& prefix, const std::string& suffix, Receivable *next ) ; + void Reset( const std::string& prefix, const std::string& suffix, DataStream *next ) ; private : static std::string Filename( const std::string& prefix, const std::string& suffix ) ; @@ -47,7 +47,7 @@ private : private : bool m_enabled ; std::ofstream m_log ; - Receivable *m_next ; + DataStream *m_next ; } ; } } // end of namespace diff --git a/libgrive/src/http/StringResponse.cc b/libgrive/src/http/StringResponse.cc index d183be3f..c316d6f3 100644 --- a/libgrive/src/http/StringResponse.cc +++ b/libgrive/src/http/StringResponse.cc @@ -30,9 +30,14 @@ void StringResponse::Clear() m_resp.clear() ; } -std::size_t StringResponse::OnData( void *data, std::size_t count ) +std::size_t StringResponse::Write( const char *data, std::size_t count ) +{ + m_resp.append( data, count ) ; + return count ; +} + +std::size_t StringResponse::Read( char *data, std::size_t count ) { - m_resp.append( reinterpret_cast(data), count ) ; return count ; } diff --git a/libgrive/src/http/StringResponse.hh b/libgrive/src/http/StringResponse.hh index e361d01c..3f262a94 100644 --- a/libgrive/src/http/StringResponse.hh +++ b/libgrive/src/http/StringResponse.hh @@ -19,18 +19,19 @@ #pragma once -#include "Receivable.hh" +#include "util/DataStream.hh" #include namespace gr { namespace http { -class StringResponse : public Receivable +class StringResponse : public DataStream { public : StringResponse() ; - std::size_t OnData( void *data, std::size_t count ) ; + std::size_t Write( const char *data, std::size_t count ) ; + std::size_t Read( char *data, std::size_t count ) ; void Clear() ; const std::string& Response() const ; diff --git a/libgrive/src/http/XmlResponse.cc b/libgrive/src/http/XmlResponse.cc index 49395f17..b25f1c4d 100644 --- a/libgrive/src/http/XmlResponse.cc +++ b/libgrive/src/http/XmlResponse.cc @@ -28,15 +28,16 @@ XmlResponse::XmlResponse() : m_tb( new xml::TreeBuilder ) { } -std::size_t XmlResponse::OnData( void *data, std::size_t count ) +std::size_t XmlResponse::Write( const char *data, std::size_t count ) { - m_tb->ParseData( reinterpret_cast(data), count ) ; + m_tb->ParseData( data, count ) ; return count ; } -void XmlResponse::Clear() +std::size_t XmlResponse::Read( char *, std::size_t ) { - m_tb.reset( new xml::TreeBuilder ) ; + // throw something better + throw -1 ; } void XmlResponse::Finish() diff --git a/libgrive/src/http/XmlResponse.hh b/libgrive/src/http/XmlResponse.hh index 6af2b745..b1505097 100644 --- a/libgrive/src/http/XmlResponse.hh +++ b/libgrive/src/http/XmlResponse.hh @@ -19,7 +19,7 @@ #pragma once -#include "Receivable.hh" +#include "util/DataStream.hh" #include @@ -31,13 +31,14 @@ namespace gr { namespace xml namespace gr { namespace http { -class XmlResponse : public Receivable +class XmlResponse : public DataStream { public : XmlResponse() ; void Clear() ; - std::size_t OnData( void *data, std::size_t count ) ; + std::size_t Write( const char *data, std::size_t count ) ; + std::size_t Read( char *data, std::size_t count ) ; void Finish() ; xml::Node Response() const ; diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index c3cdabb7..745f274c 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -48,7 +48,7 @@ Header AuthAgent::AppendHeader( const Header& hdr ) const long AuthAgent::Put( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Header auth = AppendHeader(hdr) ; @@ -62,8 +62,8 @@ long AuthAgent::Put( long AuthAgent::Put( const std::string& url, - File& file, - Receivable *dest, + File *file, + DataStream *dest, const Header& hdr ) { Header auth = AppendHeader(hdr) ; @@ -77,7 +77,7 @@ long AuthAgent::Put( long AuthAgent::Get( const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Header auth = AppendHeader(hdr) ; @@ -92,7 +92,7 @@ long AuthAgent::Get( long AuthAgent::Post( const std::string& url, const std::string& data, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Header auth = AppendHeader(hdr) ; @@ -107,7 +107,7 @@ long AuthAgent::Post( long AuthAgent::Custom( const std::string& method, const std::string& url, - Receivable *dest, + DataStream *dest, const Header& hdr ) { Header auth = AppendHeader(hdr) ; diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index fe6ae22a..ec939a8d 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -39,30 +39,30 @@ public : long Put( const std::string& url, const std::string& data, - http::Receivable *dest, + DataStream *dest, const http::Header& hdr ) ; long Put( const std::string& url, - File& file, - http::Receivable *dest, + File* file, + DataStream *dest, const http::Header& hdr ) ; long Get( const std::string& url, - http::Receivable *dest, + DataStream *dest, const http::Header& hdr ) ; long Post( const std::string& url, const std::string& data, - http::Receivable *dest, + DataStream *dest, const http::Header& hdr ) ; long Custom( const std::string& method, const std::string& url, - http::Receivable *dest, + DataStream *dest, const http::Header& hdr ) ; std::string RedirLocation() const ; diff --git a/libgrive/src/protocol/JsonResponse.cc b/libgrive/src/protocol/JsonResponse.cc index f270af6f..a30d0e32 100644 --- a/libgrive/src/protocol/JsonResponse.cc +++ b/libgrive/src/protocol/JsonResponse.cc @@ -27,14 +27,14 @@ JsonResponse::JsonResponse() { } -void JsonResponse::Clear() +std::size_t JsonResponse::Write( const char *data, std::size_t count ) { - m_resp.Clear() ; + return m_resp.Write( data, count ) ; } -std::size_t JsonResponse::OnData( void *data, std::size_t count ) +std::size_t JsonResponse::Read( char *data, std::size_t count ) { - return m_resp.OnData( data, count ) ; + return count ; } Json JsonResponse::Response() const diff --git a/libgrive/src/protocol/JsonResponse.hh b/libgrive/src/protocol/JsonResponse.hh index 3adae2da..2b6322f7 100644 --- a/libgrive/src/protocol/JsonResponse.hh +++ b/libgrive/src/protocol/JsonResponse.hh @@ -19,7 +19,7 @@ #pragma once -#include "http/Receivable.hh" +#include "util/DataStream.hh" #include "http/StringResponse.hh" namespace gr @@ -29,13 +29,13 @@ namespace gr namespace gr { namespace http { -class JsonResponse : public Receivable +class JsonResponse : public DataStream { public : JsonResponse() ; - std::size_t OnData( void *data, std::size_t count ) ; - void Clear() ; + std::size_t Write( const char *data, std::size_t count ) ; + std::size_t Read( char *data, std::size_t count ) ; Json Response() const ; diff --git a/libgrive/src/xml/NodeSet.cc b/libgrive/src/xml/NodeSet.cc index 78b74931..42d5c488 100644 --- a/libgrive/src/xml/NodeSet.cc +++ b/libgrive/src/xml/NodeSet.cc @@ -33,6 +33,10 @@ NodeSet::NodeSet() : { } +/** Initialize the node set with an external list of nodes. + With this constructor, the list of nodes pointed by [first,last) will NOT + be deep copied to the node set. +*/ NodeSet::NodeSet( iterator first, iterator last ) : m_first( first ), m_last( last ) diff --git a/libgrive/src/xml/TreeBuilder.cc b/libgrive/src/xml/TreeBuilder.cc index 3a31b65b..c1ad385a 100644 --- a/libgrive/src/xml/TreeBuilder.cc +++ b/libgrive/src/xml/TreeBuilder.cc @@ -44,6 +44,8 @@ TreeBuilder::TreeBuilder() : m_impl( new Impl ) ::XML_SetElementHandler( m_impl->psr, &TreeBuilder::StartElement, &TreeBuilder::EndElement ) ; ::XML_SetCharacterDataHandler( m_impl->psr, &TreeBuilder::OnCharData ) ; ::XML_SetUserData( m_impl->psr , this ) ; + + is_new = true ; } TreeBuilder::~TreeBuilder() @@ -68,6 +70,8 @@ Node TreeBuilder::ParseFile( const std::string& file ) void TreeBuilder::ParseData( const char *data, std::size_t count, bool last ) { + is_new = false ; + if ( ::XML_Parse( m_impl->psr, data, count, last ) == 0 ) BOOST_THROW_EXCEPTION( Error() << ExpatApiError("XML_Parse") ); } diff --git a/libgrive/src/xml/TreeBuilder.hh b/libgrive/src/xml/TreeBuilder.hh index c2a04f10..fe71a996 100644 --- a/libgrive/src/xml/TreeBuilder.hh +++ b/libgrive/src/xml/TreeBuilder.hh @@ -45,6 +45,8 @@ public : static Node ParseFile( const std::string& file ) ; static Node Parse( const std::string& xml ) ; + bool is_new ; + private : static void StartElement( void* pvthis, const char* name, const char** attr ) ; From 62dc5423417b01da522a56afc965eb9cf9f7deac Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 13:23:34 +0800 Subject: [PATCH 037/166] refactored old GDoc API into v1 namespace. using v2 namespace for new Google Drive API --- bgrive/src/main.cc | 20 ++++++-- grive/src/main.cc | 1 + libgrive/CMakeLists.txt | 3 +- libgrive/src/drive/CommonUri.cc | 4 +- libgrive/src/drive/CommonUri.hh | 4 +- libgrive/src/drive/Drive.cc | 8 +-- libgrive/src/drive/Drive.hh | 6 ++- libgrive/src/drive/Entry.cc | 4 +- libgrive/src/drive/Entry.hh | 4 +- libgrive/src/drive/Feed.cc | 4 +- libgrive/src/drive/Feed.hh | 4 +- libgrive/src/drive/Resource.cc | 6 +-- libgrive/src/drive/Resource.hh | 9 ++-- libgrive/src/drive/ResourceTree.cc | 4 +- libgrive/src/drive/ResourceTree.hh | 4 +- libgrive/src/drive/State.cc | 6 +-- libgrive/src/drive/State.hh | 5 +- libgrive/src/drive2/Drive.cc | 35 +++++++++++++ libgrive/src/drive2/Drive.hh | 45 +++++++++++++++++ libgrive/src/drive2/Feed.cc | 66 +++++++++++++++++++++++++ libgrive/src/drive2/Feed.hh | 55 +++++++++++++++++++++ libgrive/src/drive2/Resource.cc | 57 +++++++++++++++++++++ libgrive/src/drive2/Resource.hh | 49 ++++++++++++++++++ libgrive/src/protocol/Json.cc | 4 +- libgrive/test/drive/EntryTest.cc | 1 + libgrive/test/drive/ResourceTest.cc | 1 + libgrive/test/drive/ResourceTreeTest.cc | 1 + libgrive/test/drive/StateTest.cc | 1 + 28 files changed, 375 insertions(+), 36 deletions(-) create mode 100644 libgrive/src/drive2/Drive.cc create mode 100644 libgrive/src/drive2/Drive.hh create mode 100644 libgrive/src/drive2/Feed.cc create mode 100644 libgrive/src/drive2/Feed.hh create mode 100644 libgrive/src/drive2/Resource.cc create mode 100644 libgrive/src/drive2/Resource.hh diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc index 0c945f36..a45d055d 100644 --- a/bgrive/src/main.cc +++ b/bgrive/src/main.cc @@ -23,13 +23,11 @@ #include #include -#include "drive/CommonUri.hh" -#include "drive/Entry.hh" -#include "drive/Feed.hh" +#include "drive2/Feed.hh" #include "http/CurlAgent.hh" #include "http/Header.hh" -//#include "http/XmlResponse.hh" +#include "protocol/JsonResponse.hh" #include "protocol/Json.hh" #include "protocol/OAuth2.hh" @@ -38,11 +36,13 @@ #include "util/File.hh" #include +#include const std::string client_id = "22314510474.apps.googleusercontent.com" ; const std::string client_secret = "bl4ufi89h-9MkFlypcI7R785" ; using namespace gr ; +using namespace gr::v2 ; int main( int argc, char **argv ) { @@ -55,7 +55,7 @@ int main( int argc, char **argv ) OAuth2 token( refresh_token, client_id, client_secret ) ; AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; - Feed feed ; +/* Feed feed ; feed.Start( &agent, feed_base + "/-/folder?max-results=50&showroot=true" ) ; do @@ -67,6 +67,16 @@ int main( int argc, char **argv ) qDebug() << e.Name().c_str() ; } } while ( feed.GetNext( &agent ) ) ; +*/ +/* http::JsonResponse jsp ; + agent.Get( "https://www.googleapis.com/drive/v2/files", &jsp, http::Header() ) ; + std::cout << jsp.Response() << std::endl ; +*/ + Feed feed( "https://www.googleapis.com/drive/v2/files" ) ; + while ( feed.Next(&agent) ) + { + std::cout << feed.Content() << std::endl ; + } QApplication app( argc, argv ) ; MainWnd wnd ; diff --git a/grive/src/main.cc b/grive/src/main.cc index b2c43196..5209c941 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -48,6 +48,7 @@ const std::string client_id = "22314510474.apps.googleusercontent.com" ; const std::string client_secret = "bl4ufi89h-9MkFlypcI7R785" ; using namespace gr ; +using namespace gr::v1 ; namespace po = boost::program_options; // libgcrypt insist this to be done in application, not library diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index a9d6018a..08dc0660 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -45,7 +45,7 @@ include_directories( ) file(GLOB DRIVE_HEADERS - ${libgrive_SOURCE_DIR}/src/drive/*.hh + ${libgrive_SOURCE_DIR}/src/drive/*.hh ) file (GLOB PROTOCOL_HEADERS @@ -62,6 +62,7 @@ file (GLOB XML_HEADERS file (GLOB LIBGRIVE_SRC src/drive/*.cc + src/drive2/*.cc src/http/*.cc src/protocol/*.cc src/util/*.cc diff --git a/libgrive/src/drive/CommonUri.cc b/libgrive/src/drive/CommonUri.cc index dc3e0558..a69b02b4 100644 --- a/libgrive/src/drive/CommonUri.cc +++ b/libgrive/src/drive/CommonUri.cc @@ -20,7 +20,7 @@ #include "CommonUri.hh" #include -namespace gr { +namespace gr { namespace v1 { std::string ChangesFeed( int changestamp ) { @@ -28,4 +28,4 @@ std::string ChangesFeed( int changestamp ) return changestamp > 0 ? (feed%changestamp).str() : feed_changes ; } -} \ No newline at end of file +} } diff --git a/libgrive/src/drive/CommonUri.hh b/libgrive/src/drive/CommonUri.hh index 7be31b28..b0e06f7f 100644 --- a/libgrive/src/drive/CommonUri.hh +++ b/libgrive/src/drive/CommonUri.hh @@ -21,7 +21,7 @@ #include -namespace gr +namespace gr { namespace v1 { const std::string feed_base = "https://docs.google.com/feeds/default/private/full" ; const std::string feed_changes = "https://docs.google.com/feeds/default/private/changes" ; @@ -33,4 +33,4 @@ namespace gr "https://docs.google.com/feeds/upload/create-session/default/private/full" ; std::string ChangesFeed( int changestamp ) ; -} +} } diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index 4fa2cbf0..39354484 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -44,15 +44,15 @@ // for debugging only #include -namespace gr { +namespace gr { namespace v1 { namespace { const std::string state_file = ".grive_state" ; } -Drive::Drive( http::Agent *http, const Json& options ) : - m_http ( http ), +Drive::Drive( http::Agent *agent, const Json& options ) : + m_http ( agent ), m_root ( options["path"].Str() ), m_state ( m_root / state_file, options ), m_options ( options ) @@ -197,4 +197,4 @@ void Drive::UpdateChangeStamp( ) std::atoi(xrsp.Response()["docs:largestChangestamp"]["@value"].front().Value().c_str()) ) ; } -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 5424f44f..0490d5c1 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -35,12 +35,14 @@ namespace http class Agent ; } +namespace v1 { + class Entry ; class Drive { public : - Drive( http::Agent *http, const Json& options ) ; + Drive( http::Agent *agent, const Json& options ) ; void DetectChanges() ; void Update() ; @@ -64,4 +66,4 @@ private : Json m_options ; } ; -} // end of namespace +} } // end of namespace diff --git a/libgrive/src/drive/Entry.cc b/libgrive/src/drive/Entry.cc index d60d3372..23165d89 100644 --- a/libgrive/src/drive/Entry.cc +++ b/libgrive/src/drive/Entry.cc @@ -29,7 +29,7 @@ #include #include -namespace gr { +namespace gr { namespace v1 { /// construct an entry for the root folder Entry::Entry( ) : @@ -195,4 +195,4 @@ std::string Entry::Name() const return (m_kind == "file" || m_kind == "pdf") ? m_filename : m_title ; } -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Entry.hh b/libgrive/src/drive/Entry.hh index 8a9b8cc7..b7e9293a 100644 --- a/libgrive/src/drive/Entry.hh +++ b/libgrive/src/drive/Entry.hh @@ -33,6 +33,8 @@ namespace xml class Node ; } +namespace v1 { + /*! \brief corresponds to an "entry" in the resource feed This class is decodes an entry in the resource feed. It will stored the properties like @@ -95,4 +97,4 @@ private : bool m_is_removed ; } ; -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed.cc index 9c74bacc..b227e520 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed.cc @@ -29,7 +29,7 @@ #include -namespace gr { +namespace gr { namespace v1 { Feed::Feed( ) { @@ -123,4 +123,4 @@ const xml::Node& Feed::Root() const return m_root ; } -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/drive/Feed.hh index ba98a37f..92e62345 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/drive/Feed.hh @@ -36,6 +36,8 @@ namespace http class Header ; } +namespace v1 { + class Feed { public : @@ -88,4 +90,4 @@ private : reference dereference() const ; } ; -} // end of namespace +} } // end of namespace diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index b4220593..4010cab2 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -45,7 +45,7 @@ // for debugging #include -namespace gr { +namespace gr { namespace v1 { // hard coded XML file const std::string xml_meta = @@ -668,11 +668,11 @@ bool Resource::HasID() const return !m_href.empty() && !m_id.empty() ; } -} // end of namespace +} } // end of namespace namespace std { - void swap( gr::Resource& c1, gr::Resource& c2 ) + void swap( gr::v1::Resource& c1, gr::v1::Resource& c2 ) { c1.Swap( c2 ) ; } diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 5d4f0348..46096807 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -34,9 +34,12 @@ namespace http class Agent ; } -class Entry ; class Json ; +namespace v1 { + +class Entry ; + /*! \brief A resource can be a file or a folder in the google drive The google drive contains a number of resources, which is represented by this class. @@ -158,9 +161,9 @@ private : State m_state ; } ; -} // end of namespace +} } // end of namespace gr::v1 namespace std { - void swap( gr::Resource& c1, gr::Resource& c2 ) ; + void swap( gr::v1::Resource& c1, gr::v1::Resource& c2 ) ; } diff --git a/libgrive/src/drive/ResourceTree.cc b/libgrive/src/drive/ResourceTree.cc index 57615792..4a8a6daf 100644 --- a/libgrive/src/drive/ResourceTree.cc +++ b/libgrive/src/drive/ResourceTree.cc @@ -27,7 +27,7 @@ #include #include -namespace gr { +namespace gr { namespace v1 { using namespace details ; @@ -155,4 +155,4 @@ ResourceTree::iterator ResourceTree::end() return m_set.get().end() ; } -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/ResourceTree.hh b/libgrive/src/drive/ResourceTree.hh index 1edbc8c3..212141a8 100644 --- a/libgrive/src/drive/ResourceTree.hh +++ b/libgrive/src/drive/ResourceTree.hh @@ -32,6 +32,8 @@ namespace gr { class Json ; +namespace v1 { + namespace details { using namespace boost::multi_index ; @@ -96,4 +98,4 @@ private : Resource* m_root ; } ; -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 195e6967..2d1dc943 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -31,7 +31,7 @@ #include -namespace gr { +namespace gr { namespace v1 { State::State( const fs::path& filename, const Json& options ) : m_res ( options["path"].Str() ), @@ -63,7 +63,7 @@ bool State::IsIgnore( const std::string& filename ) return filename[0] == '.' ; } -void State::FromLocal( const fs::path& p, gr::Resource* folder ) +void State::FromLocal( const fs::path& p, Resource* folder ) { assert( folder != 0 ) ; assert( folder->IsFolder() ) ; @@ -294,4 +294,4 @@ void State::ChangeStamp( long cstamp ) m_cstamp = cstamp ; } -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index a0e12132..8c641892 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -34,6 +34,9 @@ namespace http } class Json ; + +namespace v1 { + class Resource ; class Entry ; @@ -80,4 +83,4 @@ private : std::vector m_unresolved ; } ; -} // end of namespace +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc new file mode 100644 index 00000000..36f28d94 --- /dev/null +++ b/libgrive/src/drive2/Drive.cc @@ -0,0 +1,35 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "Drive.hh" + +namespace gr { namespace v2 { + +Drive::Drive( ) : + m_root( "", "", "" ) +{ +} + +void Drive::Refresh( http::Agent *agent ) +{ +} + +} } // end of namespace gr::v2 + diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh new file mode 100644 index 00000000..c4e521d9 --- /dev/null +++ b/libgrive/src/drive2/Drive.hh @@ -0,0 +1,45 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include "Resource.hh" + +namespace gr { + +namespace http +{ + class Agent ; +} + +namespace v2 { + +class Drive +{ +public : + Drive( ) ; + + void Refresh( http::Agent *agent ) ; + +private : + Resource m_root ; +} ; + +} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Feed.cc b/libgrive/src/drive2/Feed.cc new file mode 100644 index 00000000..a0587abb --- /dev/null +++ b/libgrive/src/drive2/Feed.cc @@ -0,0 +1,66 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "Feed.hh" + +#include "http/Agent.hh" +#include "http/Header.hh" +#include "protocol/JsonResponse.hh" + +#include + +namespace gr { namespace v2 { + +Feed::Feed( const std::string& base ) +{ + // Next() will grab this link + m_content.Add( "nextLink", Json(base) ) ; + + Json url ; + m_content.Get("nextLink", url) ; + std::cout << "link = " << url.Str() << std::endl ; +} + +bool Feed::Next( http::Agent *agent ) +{ + Json url ; + if ( !m_content.Get("nextLink", url) ) + return false ; + + http::JsonResponse out ; + try + { + agent->Get( url.Str(), &out, http::Header() ) ; + } + catch ( Exception& e ) + { + e << DriveFeed_( m_content ) ; + throw ; + } + m_content = out.Response() ; + + return true ; +} + +Json Feed::Content() const +{ + return m_content ; +} + +} } // end of namespace diff --git a/libgrive/src/drive2/Feed.hh b/libgrive/src/drive2/Feed.hh new file mode 100644 index 00000000..dbed1af1 --- /dev/null +++ b/libgrive/src/drive2/Feed.hh @@ -0,0 +1,55 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "protocol/Json.hh" +#include "util/Exception.hh" + +#include + +namespace gr +{ +namespace http +{ + class Agent ; + class Header ; +} + +class Json ; + +namespace v2 { + +class Feed +{ +public : + // exception info + typedef boost::error_info DriveFeed_ ; + +public : + Feed( const std::string& base ) ; + bool Next( http::Agent *agent ) ; + + Json Content() const ; + +private : + Json m_content ; +} ; + +} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc new file mode 100644 index 00000000..647d0284 --- /dev/null +++ b/libgrive/src/drive2/Resource.cc @@ -0,0 +1,57 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "Resource.hh" + +namespace gr { namespace v2 { + +Resource::Resource( const std::string& id, const std::string& mime, const std::string& title ) : + m_id( id ), + m_mime( mime ), + m_title( title ) +{ +} + +std::string Resource::ID() const +{ + return m_id ; +} + +std::string Resource::Mime() const +{ + return m_mime ; +} + +std::string Resource::Title() const +{ + return m_title ; +} + +bool Resource::IsFolder() const +{ + return m_mime == "application/vnd.google-apps.folder" ; +} + +void Resource::Add( const Resource& child ) +{ + m_children.push_back( child ) ; +} + +} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.hh b/libgrive/src/drive2/Resource.hh new file mode 100644 index 00000000..41ac5519 --- /dev/null +++ b/libgrive/src/drive2/Resource.hh @@ -0,0 +1,49 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include +#include + +namespace gr { namespace v2 { + +class Resource +{ +public : + Resource( const std::string& id, const std::string& mime, const std::string& title ) ; + + std::string ID() const ; + std::string Mime() const ; + std::string Title() const ; + + bool IsFolder() const ; + + void Add( const Resource& child ) ; + +private : + std::string m_id ; + std::string m_mime ; + std::string m_title ; + + std::vector m_children ; +} ; + +} } // end of namespace gr::v2 diff --git a/libgrive/src/protocol/Json.cc b/libgrive/src/protocol/Json.cc index d290924c..4c115d6a 100644 --- a/libgrive/src/protocol/Json.cc +++ b/libgrive/src/protocol/Json.cc @@ -63,9 +63,11 @@ Json::Json( const char *str ) : ) ; } +/** Note that json_object_new_string_len() is not used. +*/ struct json_object* Json::InitStr( const char *str, std::size_t n ) { - struct json_object *j = ::json_object_new_string_len( str, n ) ; + struct json_object *j = ::json_object_new_string( str ) ; if ( j == 0 ) BOOST_THROW_EXCEPTION( Error() diff --git a/libgrive/test/drive/EntryTest.cc b/libgrive/test/drive/EntryTest.cc index 82899617..48dfa5c7 100644 --- a/libgrive/test/drive/EntryTest.cc +++ b/libgrive/test/drive/EntryTest.cc @@ -31,6 +31,7 @@ namespace grut { using namespace gr ; +using namespace gr::v1 ; EntryTest::EntryTest( ) { diff --git a/libgrive/test/drive/ResourceTest.cc b/libgrive/test/drive/ResourceTest.cc index e8190542..5f69c427 100644 --- a/libgrive/test/drive/ResourceTest.cc +++ b/libgrive/test/drive/ResourceTest.cc @@ -31,6 +31,7 @@ namespace grut { using namespace gr ; +using namespace gr::v1 ; ResourceTest::ResourceTest( ) { diff --git a/libgrive/test/drive/ResourceTreeTest.cc b/libgrive/test/drive/ResourceTreeTest.cc index 0e07d800..22285ade 100644 --- a/libgrive/test/drive/ResourceTreeTest.cc +++ b/libgrive/test/drive/ResourceTreeTest.cc @@ -29,6 +29,7 @@ namespace grut { using namespace gr ; +using namespace gr::v1 ; ResourceTreeTest::ResourceTreeTest( ) { diff --git a/libgrive/test/drive/StateTest.cc b/libgrive/test/drive/StateTest.cc index 9e33fcfa..e2ed7689 100644 --- a/libgrive/test/drive/StateTest.cc +++ b/libgrive/test/drive/StateTest.cc @@ -30,6 +30,7 @@ namespace grut { using namespace gr ; +using namespace gr::v1 ; StateTest::StateTest( ) { From 209d0b5a59ec001b4b4204e30a64256141e0e760 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 14:00:29 +0800 Subject: [PATCH 038/166] added code to fetch all folders --- bgrive/src/main.cc | 7 +++++-- libgrive/src/drive2/Drive.cc | 27 +++++++++++++++++++++++++-- libgrive/src/drive2/Drive.hh | 31 ++++++++++++++++++++++++++++++- libgrive/src/drive2/Resource.cc | 13 +++++++++++-- libgrive/src/drive2/Resource.hh | 9 +++++---- 5 files changed, 76 insertions(+), 11 deletions(-) diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc index a45d055d..fd9a0fce 100644 --- a/bgrive/src/main.cc +++ b/bgrive/src/main.cc @@ -23,7 +23,7 @@ #include #include -#include "drive2/Feed.hh" +#include "drive2/Drive.hh" #include "http/CurlAgent.hh" #include "http/Header.hh" @@ -72,11 +72,14 @@ int main( int argc, char **argv ) agent.Get( "https://www.googleapis.com/drive/v2/files", &jsp, http::Header() ) ; std::cout << jsp.Response() << std::endl ; */ - Feed feed( "https://www.googleapis.com/drive/v2/files" ) ; +/* Feed feed( "https://www.googleapis.com/drive/v2/files" ) ; while ( feed.Next(&agent) ) { std::cout << feed.Content() << std::endl ; } +*/ + Drive drive ; + drive.Refresh( &agent ) ; QApplication app( argc, argv ) ; MainWnd wnd ; diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index 36f28d94..98782536 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -20,15 +20,38 @@ #include "Drive.hh" +#include "Feed.hh" +#include "protocol/Json.hh" + +#include + namespace gr { namespace v2 { -Drive::Drive( ) : - m_root( "", "", "" ) +Drive::Drive( ) { } void Drive::Refresh( http::Agent *agent ) { + // get all folders first + Feed folders( + "https://www.googleapis.com/drive/v2/files?q=mimeType+%3d+%27application/vnd.google-apps.folder%27" ) ; + + while ( folders.Next( agent ) ) + { + std::vector items = folders.Content()["items"].AsArray() ; + for ( std::vector::iterator i = items.begin() ; i != items.end() ; ++i ) + { + const Resource *r = Add( *i ) ; + std::cout << r->Title() << " " << r->Mime() << std::endl ; + } + } +} + +const Resource* Drive::Add( const Json& item ) +{ + Resource *r = new Resource( item["id"].Str(), item["mimeType"].Str(), item["title"].Str() ) ; + return *m_db.insert(r).first ; } } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh index c4e521d9..f16a4bb7 100644 --- a/libgrive/src/drive2/Drive.hh +++ b/libgrive/src/drive2/Drive.hh @@ -22,6 +22,11 @@ #include "Resource.hh" +#include +#include +#include +#include + namespace gr { namespace http @@ -29,8 +34,29 @@ namespace http class Agent ; } +class Json ; + namespace v2 { +namespace details +{ + using namespace boost::multi_index ; + struct ByID {} ; + struct ByHref {} ; + struct ByIdentity {} ; + + typedef multi_index_container< + Resource*, + indexed_by< + hashed_unique, identity >, + hashed_non_unique, const_mem_fun > + > + > DB ; + + typedef DB::index::type ID ; + typedef DB::index::type Set ; +} + class Drive { public : @@ -39,7 +65,10 @@ public : void Refresh( http::Agent *agent ) ; private : - Resource m_root ; + const Resource* Add( const Json& item ) ; + +private : + details::DB m_db ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc index 647d0284..2c015aec 100644 --- a/libgrive/src/drive2/Resource.cc +++ b/libgrive/src/drive2/Resource.cc @@ -22,6 +22,15 @@ namespace gr { namespace v2 { +/** Default constructor construct the resource of the root folder +*/ +Resource::Resource() : + m_id( "root" ), + m_mime( "application/vnd.google-apps.folder" ), + m_title( "Root folder" ) +{ +} + Resource::Resource( const std::string& id, const std::string& mime, const std::string& title ) : m_id( id ), m_mime( mime ), @@ -49,9 +58,9 @@ bool Resource::IsFolder() const return m_mime == "application/vnd.google-apps.folder" ; } -void Resource::Add( const Resource& child ) +void Resource::Add( const std::string& child_id ) { - m_children.push_back( child ) ; + m_children.push_back( child_id ) ; } } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.hh b/libgrive/src/drive2/Resource.hh index 41ac5519..f893e0c5 100644 --- a/libgrive/src/drive2/Resource.hh +++ b/libgrive/src/drive2/Resource.hh @@ -28,6 +28,7 @@ namespace gr { namespace v2 { class Resource { public : + Resource() ; Resource( const std::string& id, const std::string& mime, const std::string& title ) ; std::string ID() const ; @@ -35,15 +36,15 @@ public : std::string Title() const ; bool IsFolder() const ; - - void Add( const Resource& child ) ; - + + void Add( const std::string& child_id ) ; + private : std::string m_id ; std::string m_mime ; std::string m_title ; - std::vector m_children ; + std::vector m_children ; } ; } } // end of namespace gr::v2 From 87d96972f7ceaa1a2bf110892a7cfaceafca2548 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 14:50:54 +0800 Subject: [PATCH 039/166] Added Query() helper to Feed. and common URI header --- libgrive/src/drive2/CommonUri.hh | 36 ++++++++++++++++++++++++++++++++ libgrive/src/drive2/Drive.cc | 17 +++++++++++---- libgrive/src/drive2/Drive.hh | 2 ++ libgrive/src/drive2/Feed.cc | 10 +++++---- libgrive/src/drive2/Feed.hh | 3 +++ libgrive/src/drive2/Resource.cc | 4 +++- 6 files changed, 63 insertions(+), 9 deletions(-) create mode 100644 libgrive/src/drive2/CommonUri.hh diff --git a/libgrive/src/drive2/CommonUri.hh b/libgrive/src/drive2/CommonUri.hh new file mode 100644 index 00000000..19cc32cd --- /dev/null +++ b/libgrive/src/drive2/CommonUri.hh @@ -0,0 +1,36 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include + +namespace gr { namespace v2 { + +namespace feeds +{ + const std::string files = "https://www.googleapis.com/drive/v2/files" ; +} + +namespace mime_types +{ + const std::string folder = "application/vnd.google-apps.folder" ; +} + +} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index 98782536..cdfdb766 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -20,6 +20,7 @@ #include "Drive.hh" +#include "CommonUri.hh" #include "Feed.hh" #include "protocol/Json.hh" @@ -34,16 +35,18 @@ Drive::Drive( ) void Drive::Refresh( http::Agent *agent ) { // get all folders first - Feed folders( - "https://www.googleapis.com/drive/v2/files?q=mimeType+%3d+%27application/vnd.google-apps.folder%27" ) ; - +// Feed folders( +// "https://www.googleapis.com/drive/v2/files?q=mimeType+%3d+%27application/vnd.google-apps.folder%27" ) ; + Feed folders( feeds::files ) ; +// folders.Query( "mimeType", "application/vnd.google-apps.folder" ) ; + while ( folders.Next( agent ) ) { std::vector items = folders.Content()["items"].AsArray() ; for ( std::vector::iterator i = items.begin() ; i != items.end() ; ++i ) { const Resource *r = Add( *i ) ; - std::cout << r->Title() << " " << r->Mime() << std::endl ; + std::cout << r->Title() << " " << r->Mime() << std::endl ; } } } @@ -54,5 +57,11 @@ const Resource* Drive::Add( const Json& item ) return *m_db.insert(r).first ; } +Resource* Drive::Find( const std::string& id ) +{ + details::ID::iterator i = m_db.get().find(id) ; + return i != m_db.get().end() ? *i : 0 ; +} + } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh index f16a4bb7..241ab7f9 100644 --- a/libgrive/src/drive2/Drive.hh +++ b/libgrive/src/drive2/Drive.hh @@ -64,6 +64,8 @@ public : void Refresh( http::Agent *agent ) ; + Resource* Find( const std::string& id ) ; + private : const Resource* Add( const Json& item ) ; diff --git a/libgrive/src/drive2/Feed.cc b/libgrive/src/drive2/Feed.cc index a0587abb..dd59ca5c 100644 --- a/libgrive/src/drive2/Feed.cc +++ b/libgrive/src/drive2/Feed.cc @@ -31,10 +31,12 @@ Feed::Feed( const std::string& base ) { // Next() will grab this link m_content.Add( "nextLink", Json(base) ) ; - - Json url ; - m_content.Get("nextLink", url) ; - std::cout << "link = " << url.Str() << std::endl ; +} + +void Feed::Query( const std::string& field, const std::string& value ) +{ + std::string url = m_content["nextLink"].Str() ; + m_content.Add( "nextLink", Json( url + "?q=" + field + "+%3d+%27" + value + "%27" ) ) ; } bool Feed::Next( http::Agent *agent ) diff --git a/libgrive/src/drive2/Feed.hh b/libgrive/src/drive2/Feed.hh index dbed1af1..157f1e37 100644 --- a/libgrive/src/drive2/Feed.hh +++ b/libgrive/src/drive2/Feed.hh @@ -44,6 +44,9 @@ public : public : Feed( const std::string& base ) ; + void Query( const std::string& field, const std::string& value ) ; + + bool Next( http::Agent *agent ) ; Json Content() const ; diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc index 2c015aec..1e33a586 100644 --- a/libgrive/src/drive2/Resource.cc +++ b/libgrive/src/drive2/Resource.cc @@ -20,13 +20,15 @@ #include "Resource.hh" +#include "CommonUri.hh" + namespace gr { namespace v2 { /** Default constructor construct the resource of the root folder */ Resource::Resource() : m_id( "root" ), - m_mime( "application/vnd.google-apps.folder" ), + m_mime( mime_types::folder ), m_title( "Root folder" ) { } From ee6408d05f2c4577d36f4ccac71150df99df3447 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 15:45:58 +0800 Subject: [PATCH 040/166] changed to store parent IDs --- libgrive/src/drive2/CommonUri.hh | 5 +++++ libgrive/src/drive2/Drive.cc | 14 +++++++++++++- libgrive/src/drive2/Resource.cc | 5 ----- libgrive/src/drive2/Resource.hh | 12 ++++++++---- libgrive/src/protocol/Json.hh | 12 ++++++++++++ 5 files changed, 38 insertions(+), 10 deletions(-) diff --git a/libgrive/src/drive2/CommonUri.hh b/libgrive/src/drive2/CommonUri.hh index 19cc32cd..8f5304c1 100644 --- a/libgrive/src/drive2/CommonUri.hh +++ b/libgrive/src/drive2/CommonUri.hh @@ -33,4 +33,9 @@ namespace mime_types const std::string folder = "application/vnd.google-apps.folder" ; } +namespace kinds +{ + const std::string parent = "drive#parentReference" ; +} + } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index cdfdb766..e731ef21 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -25,6 +25,7 @@ #include "protocol/Json.hh" #include +#include namespace gr { namespace v2 { @@ -46,7 +47,6 @@ void Drive::Refresh( http::Agent *agent ) for ( std::vector::iterator i = items.begin() ; i != items.end() ; ++i ) { const Resource *r = Add( *i ) ; - std::cout << r->Title() << " " << r->Mime() << std::endl ; } } } @@ -54,6 +54,18 @@ void Drive::Refresh( http::Agent *agent ) const Resource* Drive::Add( const Json& item ) { Resource *r = new Resource( item["id"].Str(), item["mimeType"].Str(), item["title"].Str() ) ; + + // initialize parent IDs + Json parents ; + if ( item.Get( "parents", parents ) ) + { + std::vector parent_ids ; + parents.Select( "id", std::back_inserter(parent_ids) ) ; + + r->SetParent( parent_ids.begin(), parent_ids.end() ) ; + std::cout << r->Title() << " " << r->ID() << " " << parent_ids.size() << std::endl ; + } + return *m_db.insert(r).first ; } diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc index 1e33a586..6724d1e9 100644 --- a/libgrive/src/drive2/Resource.cc +++ b/libgrive/src/drive2/Resource.cc @@ -60,9 +60,4 @@ bool Resource::IsFolder() const return m_mime == "application/vnd.google-apps.folder" ; } -void Resource::Add( const std::string& child_id ) -{ - m_children.push_back( child_id ) ; -} - } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.hh b/libgrive/src/drive2/Resource.hh index f893e0c5..ca4a599b 100644 --- a/libgrive/src/drive2/Resource.hh +++ b/libgrive/src/drive2/Resource.hh @@ -30,21 +30,25 @@ class Resource public : Resource() ; Resource( const std::string& id, const std::string& mime, const std::string& title ) ; - + + template + void SetParent( InputIt first, InputIt last ) + { + m_parent.assign( first, last ) ; + } + std::string ID() const ; std::string Mime() const ; std::string Title() const ; bool IsFolder() const ; - void Add( const std::string& child_id ) ; - private : std::string m_id ; std::string m_mime ; std::string m_title ; - std::vector m_children ; + std::vector m_parent ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/protocol/Json.hh b/libgrive/src/protocol/Json.hh index 1ebf1737..be6ff6fa 100644 --- a/libgrive/src/protocol/Json.hh +++ b/libgrive/src/protocol/Json.hh @@ -100,6 +100,18 @@ public : Json FindInArray( const std::string& key, const std::string& value ) const ; bool FindInArray( const std::string& key, const std::string& value, Json& result ) const ; + /** Expect *this is a JSON array of objects. Select all "key" values inside each + objects in the array and copies them in the output iterator \a out. + */ + template + Out Select( const std::string& key, Out out ) + { + Array a = AsArray() ; + for ( Array::iterator i = a.begin() ; i != a.end() ; ++i ) + *out++ = i->As() ; + return out ; + } + friend std::ostream& operator<<( std::ostream& os, const Json& json ) ; void Write( DataStream *out ) const ; From 326ae20ca72d2380882e7237f5da323b75e121c9 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 17:35:09 +0800 Subject: [PATCH 041/166] showing in qt model, but no child --- bgrive/src/DriveModel.cc | 32 ++++++++++--- bgrive/src/DriveModel.hh | 15 ++++++- bgrive/src/MainWnd.cc | 3 +- bgrive/src/MainWnd.hh | 7 ++- bgrive/src/main.cc | 6 +-- libgrive/src/drive2/Drive.cc | 80 ++++++++++++++++++++++++++------- libgrive/src/drive2/Drive.hh | 13 +++++- libgrive/src/drive2/Resource.cc | 15 +++++++ libgrive/src/drive2/Resource.hh | 14 +++--- 9 files changed, 150 insertions(+), 35 deletions(-) diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc index e207fb4f..8d652713 100644 --- a/bgrive/src/DriveModel.cc +++ b/bgrive/src/DriveModel.cc @@ -20,12 +20,17 @@ #include "DriveModel.hh" +#include "drive2/Resource.hh" + #include namespace gr { -DriveModel::DriveModel( ) +using namespace v2; + +DriveModel::DriveModel( http::Agent *agent ) { + m_drv.Refresh( agent ) ; } Qt::ItemFlags DriveModel::flags( const QModelIndex& ) const @@ -35,7 +40,17 @@ Qt::ItemFlags DriveModel::flags( const QModelIndex& ) const QVariant DriveModel::data( const QModelIndex& index, int role ) const { - return role == Qt::DisplayRole ? QString("wow") : QVariant() ; + const Resource *res = Res(index) ; + if ( role == Qt::DisplayRole && res != 0 ) + { + switch ( index.column() ) + { + case 0: return QString::fromUtf8(res->Title().c_str()) ; + default: break ; + } + } + + return QVariant() ; } QVariant DriveModel::headerData( int section, Qt::Orientation orientation, int role ) const @@ -45,7 +60,7 @@ QVariant DriveModel::headerData( int section, Qt::Orientation orientation, int r int DriveModel::rowCount( const QModelIndex& parent ) const { - return 10 ; + return Res(parent)->ChildCount() ; } int DriveModel::columnCount( const QModelIndex& parent ) const @@ -55,12 +70,19 @@ int DriveModel::columnCount( const QModelIndex& parent ) const bool DriveModel::hasChildren( const QModelIndex& parent ) const { - return parent.isValid() ? false : true ; + return Res(parent)->ChildCount() > 0 ; } QModelIndex DriveModel::index( int row, int column, const QModelIndex & parent ) const { - return parent.isValid() ? QModelIndex() : createIndex( row, column, 0 ) ; + return createIndex( row, column, const_cast(m_drv.Child(Res(parent), row)) ) ; +} + +const Resource* DriveModel::Res( const QModelIndex& idx ) const +{ + return idx.isValid() + ? reinterpret_cast(idx.internalPointer()) + : m_drv.Root() ; } QModelIndex DriveModel::parent( const QModelIndex& idx ) const diff --git a/bgrive/src/DriveModel.hh b/bgrive/src/DriveModel.hh index f7ee26ff..1a74283c 100644 --- a/bgrive/src/DriveModel.hh +++ b/bgrive/src/DriveModel.hh @@ -22,12 +22,19 @@ #include +#include "drive2/Drive.hh" + namespace gr { +namespace http +{ + class Agent ; +} + class DriveModel : public QAbstractItemModel { public : - DriveModel( ) ; + DriveModel( http::Agent *agent ) ; // QAbstractItemModel overrides Qt::ItemFlags flags( const QModelIndex & index ) const ; @@ -38,6 +45,12 @@ public : bool hasChildren ( const QModelIndex& parent ) const ; QModelIndex index( int row, int column, const QModelIndex& parent ) const ; QModelIndex parent( const QModelIndex& idx ) const ; + +private : + const v2::Resource* Res( const QModelIndex& idx ) const ; + +private : + v2::Drive m_drv ; } ; } // end of namespace diff --git a/bgrive/src/MainWnd.cc b/bgrive/src/MainWnd.cc index d73c9c95..84bf64fa 100644 --- a/bgrive/src/MainWnd.cc +++ b/bgrive/src/MainWnd.cc @@ -24,7 +24,8 @@ namespace gr { -MainWnd::MainWnd( ) +MainWnd::MainWnd( http::Agent *agent ) : + m_drive( agent ) { m_ui.setupUi(this) ; diff --git a/bgrive/src/MainWnd.hh b/bgrive/src/MainWnd.hh index d5be6644..5551a6b6 100644 --- a/bgrive/src/MainWnd.hh +++ b/bgrive/src/MainWnd.hh @@ -27,12 +27,17 @@ namespace gr { +namespace http +{ + class Agent ; +} + class MainWnd : public QMainWindow { Q_OBJECT public : - MainWnd( ) ; + MainWnd( http::Agent *agent ) ; private : Ui::MainWindow m_ui ; diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc index fd9a0fce..dd4bd939 100644 --- a/bgrive/src/main.cc +++ b/bgrive/src/main.cc @@ -78,11 +78,11 @@ int main( int argc, char **argv ) std::cout << feed.Content() << std::endl ; } */ - Drive drive ; - drive.Refresh( &agent ) ; +// Drive drive ; +// drive.Refresh( &agent ) ; QApplication app( argc, argv ) ; - MainWnd wnd ; + MainWnd wnd( &agent ) ; wnd.show(); return app.exec() ; diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index e731ef21..9267be0c 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -27,6 +27,8 @@ #include #include +#include + namespace gr { namespace v2 { Drive::Drive( ) @@ -36,37 +38,64 @@ Drive::Drive( ) void Drive::Refresh( http::Agent *agent ) { // get all folders first -// Feed folders( -// "https://www.googleapis.com/drive/v2/files?q=mimeType+%3d+%27application/vnd.google-apps.folder%27" ) ; Feed folders( feeds::files ) ; -// folders.Query( "mimeType", "application/vnd.google-apps.folder" ) ; + std::vector > parent_child ; while ( folders.Next( agent ) ) { std::vector items = folders.Content()["items"].AsArray() ; for ( std::vector::iterator i = items.begin() ; i != items.end() ; ++i ) { - const Resource *r = Add( *i ) ; + Resource *r = NewResource( *i ) ; + + Json parents ; + if ( i->Get( "parents", parents ) ) + { + std::vector pids ; + parents.Select( "id", std::back_inserter(pids) ) ; + + for ( std::vector::iterator p = pids.begin() ; p != pids.end() ; ++p ) + parent_child.push_back( std::make_pair( *p, r ) ) ; + + // add to root node if no parent + if ( pids.empty() ) + m_root.AddChild( r->ID() ) ; + } } } + + for ( std::vector >::iterator i = parent_child.begin() ; + i != parent_child.end() ; ++i ) + { + Resource *parent = Find( i->first ) ; + if ( parent != 0 ) + parent->AddChild( i->second->ID() ) ; + } } -const Resource* Drive::Add( const Json& item ) +Resource* Drive::FindRoot( http::Agent *agent ) +{ + // get all folders first + Feed folders( feeds::files + "/root?fields=id" ) ; + folders.Next( agent ) ; + + std::string id = folders.Content()["id"].Str() ; + std::cout << "root = " << id << std::endl ; + + return Find( folders.Content()["id"].Str() ) ; +} + +Resource* Drive::NewResource( const Json& item ) { Resource *r = new Resource( item["id"].Str(), item["mimeType"].Str(), item["title"].Str() ) ; // initialize parent IDs - Json parents ; - if ( item.Get( "parents", parents ) ) - { - std::vector parent_ids ; - parents.Select( "id", std::back_inserter(parent_ids) ) ; - - r->SetParent( parent_ids.begin(), parent_ids.end() ) ; - std::cout << r->Title() << " " << r->ID() << " " << parent_ids.size() << std::endl ; - } + std::cout << r->Title() << " " << r->ID() << std::endl ; + + m_db.insert(r) ; + assert( Find(r->ID()) == r ) ; - return *m_db.insert(r).first ; + return r ; } Resource* Drive::Find( const std::string& id ) @@ -75,5 +104,26 @@ Resource* Drive::Find( const std::string& id ) return i != m_db.get().end() ? *i : 0 ; } +const Resource* Drive::Find( const std::string& id ) const +{ + details::ID::const_iterator i = m_db.get().find(id) ; + return i != m_db.get().end() ? *i : 0 ; +} + +Resource* Drive::Root() +{ + return &m_root ; +} + +const Resource* Drive::Root() const +{ + return &m_root ; +} + +const Resource* Drive::Child( const Resource *parent, std::size_t idx ) const +{ + return Find( parent->At(idx) ) ; +} + } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh index 241ab7f9..ba5ecd5e 100644 --- a/libgrive/src/drive2/Drive.hh +++ b/libgrive/src/drive2/Drive.hh @@ -65,12 +65,21 @@ public : void Refresh( http::Agent *agent ) ; Resource* Find( const std::string& id ) ; + const Resource* Find( const std::string& id ) const ; -private : - const Resource* Add( const Json& item ) ; + Resource* Root() ; + const Resource* Root() const ; + + const Resource* Child( const Resource *parent, std::size_t idx ) const ; +private : + Resource* NewResource( const Json& item ) ; + Resource* FindRoot( http::Agent *agent ) ; + private : details::DB m_db ; + + Resource m_root ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc index 6724d1e9..3db446b0 100644 --- a/libgrive/src/drive2/Resource.cc +++ b/libgrive/src/drive2/Resource.cc @@ -60,4 +60,19 @@ bool Resource::IsFolder() const return m_mime == "application/vnd.google-apps.folder" ; } +void Resource::AddChild( const std::string& child ) +{ + m_children.push_back( child ) ; +} + +std::size_t Resource::ChildCount() const +{ + return m_children.size() ; +} + +std::string Resource::At( std::size_t idx ) const +{ + return m_children.at(idx) ; +} + } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.hh b/libgrive/src/drive2/Resource.hh index ca4a599b..ccfc477c 100644 --- a/libgrive/src/drive2/Resource.hh +++ b/libgrive/src/drive2/Resource.hh @@ -31,24 +31,24 @@ public : Resource() ; Resource( const std::string& id, const std::string& mime, const std::string& title ) ; - template - void SetParent( InputIt first, InputIt last ) - { - m_parent.assign( first, last ) ; - } - std::string ID() const ; std::string Mime() const ; std::string Title() const ; bool IsFolder() const ; + void AddChild( const std::string& child ) ; + + std::size_t ChildCount() const ; + + std::string At( std::size_t idx ) const ; + private : std::string m_id ; std::string m_mime ; std::string m_title ; - std::vector m_parent ; + std::vector m_children ; } ; } } // end of namespace gr::v2 From 94d66f0d30e2a46fd313b0e86fc2927692c4a970 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 22:02:41 +0800 Subject: [PATCH 042/166] can show tree now, but still has issues --- bgrive/src/DriveModel.cc | 15 ++++++++- libgrive/src/drive2/Drive.cc | 54 +++++++++++++++++++-------------- libgrive/src/drive2/Drive.hh | 5 +-- libgrive/src/drive2/Resource.cc | 17 +++++++++++ libgrive/src/drive2/Resource.hh | 6 +++- libgrive/src/protocol/Json.hh | 6 +++- 6 files changed, 76 insertions(+), 27 deletions(-) diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc index 8d652713..c58f757d 100644 --- a/bgrive/src/DriveModel.cc +++ b/bgrive/src/DriveModel.cc @@ -70,6 +70,7 @@ int DriveModel::columnCount( const QModelIndex& parent ) const bool DriveModel::hasChildren( const QModelIndex& parent ) const { +qDebug() << Res(parent)->Title().c_str() << " has " << Res(parent)->ChildCount() << " children" ; return Res(parent)->ChildCount() > 0 ; } @@ -87,7 +88,19 @@ const Resource* DriveModel::Res( const QModelIndex& idx ) const QModelIndex DriveModel::parent( const QModelIndex& idx ) const { - return QModelIndex() ; + // if I am root, my parent is myself + const Resource *res = Res(idx) ; + if ( res == m_drv.Root() ) + return QModelIndex() ; + + // if my parent is root, return model index of root (i.e. QModelIndex()) + const Resource *parent = m_drv.Parent(res) ; + if ( parent == 0 || parent == m_drv.Root() || idx.column() != 0 ) + return QModelIndex() ; + + // check grand-parent to know the row() of my parent + const Resource *grand = m_drv.Parent(parent) ; + return createIndex( grand->Index(parent->ID()), 0, const_cast(grand) ) ; } } // end of namespace diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index 9267be0c..97e9ad50 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -31,23 +31,30 @@ namespace gr { namespace v2 { -Drive::Drive( ) +Drive::Drive( ) : + m_root( 0 ) { + } void Drive::Refresh( http::Agent *agent ) { + // find root node ID + assert( m_root == 0 ) ; + m_root = NewResource( agent, "root" ) ; + // get all folders first Feed folders( feeds::files ) ; + folders.Query( "mimeType", mime_types::folder ) ; std::vector > parent_child ; - while ( folders.Next( agent ) ) { std::vector items = folders.Content()["items"].AsArray() ; for ( std::vector::iterator i = items.begin() ; i != items.end() ; ++i ) { Resource *r = NewResource( *i ) ; - +//std::cout << r->Title() << " " << r->ID() << std::endl ; + Json parents ; if ( i->Get( "parents", parents ) ) { @@ -55,11 +62,10 @@ void Drive::Refresh( http::Agent *agent ) parents.Select( "id", std::back_inserter(pids) ) ; for ( std::vector::iterator p = pids.begin() ; p != pids.end() ; ++p ) + { + std::cout << "parent = " << *p << std::endl ; parent_child.push_back( std::make_pair( *p, r ) ) ; - - // add to root node if no parent - if ( pids.empty() ) - m_root.AddChild( r->ID() ) ; + } } } } @@ -67,31 +73,30 @@ void Drive::Refresh( http::Agent *agent ) for ( std::vector >::iterator i = parent_child.begin() ; i != parent_child.end() ; ++i ) { - Resource *parent = Find( i->first ) ; + Resource *parent = Find( i->first ), *child = i->second ; +std::cout << i->first << " " << child->ID() << std::endl ; if ( parent != 0 ) - parent->AddChild( i->second->ID() ) ; + { +// initialize parent IDs + + parent->AddChild( child->ID() ) ; + } } } -Resource* Drive::FindRoot( http::Agent *agent ) +/// Create resource base on ID +Resource* Drive::NewResource( http::Agent *agent, const std::string& id ) { - // get all folders first - Feed folders( feeds::files + "/root?fields=id" ) ; - folders.Next( agent ) ; - - std::string id = folders.Content()["id"].Str() ; - std::cout << "root = " << id << std::endl ; + Feed feed( feeds::files + "/" + id ) ; + feed.Next( agent ) ; - return Find( folders.Content()["id"].Str() ) ; + return NewResource( feed.Content() ) ; } Resource* Drive::NewResource( const Json& item ) { Resource *r = new Resource( item["id"].Str(), item["mimeType"].Str(), item["title"].Str() ) ; - // initialize parent IDs - std::cout << r->Title() << " " << r->ID() << std::endl ; - m_db.insert(r) ; assert( Find(r->ID()) == r ) ; @@ -112,12 +117,12 @@ const Resource* Drive::Find( const std::string& id ) const Resource* Drive::Root() { - return &m_root ; + return m_root ; } const Resource* Drive::Root() const { - return &m_root ; + return m_root ; } const Resource* Drive::Child( const Resource *parent, std::size_t idx ) const @@ -125,5 +130,10 @@ const Resource* Drive::Child( const Resource *parent, std::size_t idx ) const return Find( parent->At(idx) ) ; } +const Resource* Drive::Parent( const Resource *child ) const +{ + return Find( child->Parent() ) ; +} + } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh index ba5ecd5e..3c9b8569 100644 --- a/libgrive/src/drive2/Drive.hh +++ b/libgrive/src/drive2/Drive.hh @@ -71,15 +71,16 @@ public : const Resource* Root() const ; const Resource* Child( const Resource *parent, std::size_t idx ) const ; + const Resource* Parent( const Resource *child ) const ; private : Resource* NewResource( const Json& item ) ; - Resource* FindRoot( http::Agent *agent ) ; + Resource* NewResource( http::Agent *agent, const std::string& id ) ; private : details::DB m_db ; - Resource m_root ; + Resource *m_root ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc index 3db446b0..40e681b5 100644 --- a/libgrive/src/drive2/Resource.cc +++ b/libgrive/src/drive2/Resource.cc @@ -22,6 +22,8 @@ #include "CommonUri.hh" +#include + namespace gr { namespace v2 { /** Default constructor construct the resource of the root folder @@ -65,6 +67,11 @@ void Resource::AddChild( const std::string& child ) m_children.push_back( child ) ; } +void Resource::SetParent( const std::string& parent ) +{ + m_parent = parent ; +} + std::size_t Resource::ChildCount() const { return m_children.size() ; @@ -75,4 +82,14 @@ std::string Resource::At( std::size_t idx ) const return m_children.at(idx) ; } +std::string Resource::Parent() const +{ + return m_parent ; +} + +std::size_t Resource::Index( const std::string& child ) const +{ + return std::find( m_children.begin(), m_children.end(), child ) - m_children.begin() ; +} + } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.hh b/libgrive/src/drive2/Resource.hh index ccfc477c..7a38275d 100644 --- a/libgrive/src/drive2/Resource.hh +++ b/libgrive/src/drive2/Resource.hh @@ -38,10 +38,12 @@ public : bool IsFolder() const ; void AddChild( const std::string& child ) ; + void SetParent( const std::string& parent ) ; std::size_t ChildCount() const ; - std::string At( std::size_t idx ) const ; + std::string Parent() const ; + std::size_t Index( const std::string& child ) const ; private : std::string m_id ; @@ -49,6 +51,8 @@ private : std::string m_title ; std::vector m_children ; + + std::string m_parent ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/protocol/Json.hh b/libgrive/src/protocol/Json.hh index be6ff6fa..320f9d71 100644 --- a/libgrive/src/protocol/Json.hh +++ b/libgrive/src/protocol/Json.hh @@ -108,7 +108,11 @@ public : { Array a = AsArray() ; for ( Array::iterator i = a.begin() ; i != a.end() ; ++i ) - *out++ = i->As() ; + { + Json value; + if ( i->Get( key, value ) ) + *out++ = value.As() ; + } return out ; } From 93fe68ce4c225179111daa8b88218636f64595bf Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 22:14:05 +0800 Subject: [PATCH 043/166] fixing the problems with multiple parent --- bgrive/src/DriveModel.cc | 10 +++++++--- libgrive/src/drive2/Drive.cc | 21 ++++++++++++++------- 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc index c58f757d..65b80909 100644 --- a/bgrive/src/DriveModel.cc +++ b/bgrive/src/DriveModel.cc @@ -70,7 +70,6 @@ int DriveModel::columnCount( const QModelIndex& parent ) const bool DriveModel::hasChildren( const QModelIndex& parent ) const { -qDebug() << Res(parent)->Title().c_str() << " has " << Res(parent)->ChildCount() << " children" ; return Res(parent)->ChildCount() > 0 ; } @@ -93,14 +92,19 @@ QModelIndex DriveModel::parent( const QModelIndex& idx ) const if ( res == m_drv.Root() ) return QModelIndex() ; + qDebug() << "getting parent of " << res->Title().c_str() ; + // if my parent is root, return model index of root (i.e. QModelIndex()) const Resource *parent = m_drv.Parent(res) ; + if ( parent == 0 || parent == m_drv.Root() || idx.column() != 0 ) return QModelIndex() ; - + + qDebug() << "parent of " << res->Title().c_str() << " should be " << parent->Title().c_str(); + // check grand-parent to know the row() of my parent const Resource *grand = m_drv.Parent(parent) ; - return createIndex( grand->Index(parent->ID()), 0, const_cast(grand) ) ; + return createIndex( grand->Index(parent->ID()), 0, const_cast(parent) ) ; } } // end of namespace diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index 97e9ad50..945c78a6 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -60,11 +60,15 @@ void Drive::Refresh( http::Agent *agent ) { std::vector pids ; parents.Select( "id", std::back_inserter(pids) ) ; - - for ( std::vector::iterator p = pids.begin() ; p != pids.end() ; ++p ) + + // onlly the first parent counts + if ( !pids.empty() ) + parent_child.push_back( std::make_pair( pids.front(), r ) ) ; + +// for ( std::vector::iterator p = pids.begin() ; p != pids.end() ; ++p ) { - std::cout << "parent = " << *p << std::endl ; - parent_child.push_back( std::make_pair( *p, r ) ) ; +// std::cout << "parent = " << *p << std::endl ; +// parent_child.push_back( std::make_pair( *p, r ) ) ; } } } @@ -74,12 +78,15 @@ void Drive::Refresh( http::Agent *agent ) i != parent_child.end() ; ++i ) { Resource *parent = Find( i->first ), *child = i->second ; -std::cout << i->first << " " << child->ID() << std::endl ; + assert( child != 0 ) ; + if ( parent != 0 ) { -// initialize parent IDs - + std::cout << "parent of " << child->Title() << " is " << parent->Title() << std::endl ; + + // initialize parent IDs parent->AddChild( child->ID() ) ; + child->SetParent( parent->ID() ) ; } } } From bba65c0cc719c46de70f730bccb21f697ee96ab3 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 22:31:12 +0800 Subject: [PATCH 044/166] show files now. working good. --- bgrive/src/DriveModel.cc | 4 -- libgrive/src/drive2/Drive.cc | 72 ++++++++++++++++++--------------- libgrive/src/drive2/Drive.hh | 3 ++ libgrive/src/drive2/Resource.cc | 13 ++++-- libgrive/src/drive2/Resource.hh | 6 ++- 5 files changed, 56 insertions(+), 42 deletions(-) diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc index 65b80909..e2b7d4b0 100644 --- a/bgrive/src/DriveModel.cc +++ b/bgrive/src/DriveModel.cc @@ -92,16 +92,12 @@ QModelIndex DriveModel::parent( const QModelIndex& idx ) const if ( res == m_drv.Root() ) return QModelIndex() ; - qDebug() << "getting parent of " << res->Title().c_str() ; - // if my parent is root, return model index of root (i.e. QModelIndex()) const Resource *parent = m_drv.Parent(res) ; if ( parent == 0 || parent == m_drv.Root() || idx.column() != 0 ) return QModelIndex() ; - qDebug() << "parent of " << res->Title().c_str() << " should be " << parent->Title().c_str(); - // check grand-parent to know the row() of my parent const Resource *grand = m_drv.Parent(parent) ; return createIndex( grand->Index(parent->ID()), 0, const_cast(parent) ) ; diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index 945c78a6..2a28cfd0 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -46,44 +46,20 @@ void Drive::Refresh( http::Agent *agent ) // get all folders first Feed folders( feeds::files ) ; folders.Query( "mimeType", mime_types::folder ) ; - std::vector > parent_child ; - while ( folders.Next( agent ) ) - { - std::vector items = folders.Content()["items"].AsArray() ; - for ( std::vector::iterator i = items.begin() ; i != items.end() ; ++i ) - { - Resource *r = NewResource( *i ) ; -//std::cout << r->Title() << " " << r->ID() << std::endl ; - - Json parents ; - if ( i->Get( "parents", parents ) ) - { - std::vector pids ; - parents.Select( "id", std::back_inserter(pids) ) ; - - // onlly the first parent counts - if ( !pids.empty() ) - parent_child.push_back( std::make_pair( pids.front(), r ) ) ; - -// for ( std::vector::iterator p = pids.begin() ; p != pids.end() ; ++p ) - { -// std::cout << "parent = " << *p << std::endl ; -// parent_child.push_back( std::make_pair( *p, r ) ) ; - } - } - } - } + NewResource( agent, folders ) ; - for ( std::vector >::iterator i = parent_child.begin() ; - i != parent_child.end() ; ++i ) + // get all files + Feed files( feeds::files ) ; + NewResource( agent, files ) ; + + // build parent-child linkage between folders + for ( details::DB::iterator i = m_db.begin() ; i != m_db.end() ; ++i ) { - Resource *parent = Find( i->first ), *child = i->second ; + Resource *parent = Find( (*i)->Parent() ), *child = *i ; assert( child != 0 ) ; if ( parent != 0 ) { - std::cout << "parent of " << child->Title() << " is " << parent->Title() << std::endl ; - // initialize parent IDs parent->AddChild( child->ID() ) ; child->SetParent( parent->ID() ) ; @@ -91,6 +67,18 @@ void Drive::Refresh( http::Agent *agent ) } } +void Drive::NewResource( http::Agent *agent, Feed& items ) +{ + assert( agent != 0 ) ; + + while ( items.Next( agent ) ) + { + std::vector item_json = items.Content()["items"].AsArray() ; + for ( std::vector::iterator i = item_json.begin() ; i != item_json.end() ; ++i ) + NewResource( *i ) ; + } +} + /// Create resource base on ID Resource* Drive::NewResource( http::Agent *agent, const std::string& id ) { @@ -102,7 +90,25 @@ Resource* Drive::NewResource( http::Agent *agent, const std::string& id ) Resource* Drive::NewResource( const Json& item ) { - Resource *r = new Resource( item["id"].Str(), item["mimeType"].Str(), item["title"].Str() ) ; + // assume resource is directly under root + std::string parent_id = m_root != 0 ? m_root->ID() : "" ; + + Json parents ; + if ( item.Get( "parents", parents ) ) + { + std::vector pids ; + parents.Select( "id", std::back_inserter(pids) ) ; + + // only the first parent counts + if ( !pids.empty() ) + parent_id = pids.front() ; + } + + Resource *r = new Resource( + item["id"].Str(), + item["mimeType"].Str(), + item["title"].Str(), + parent_id ) ; m_db.insert(r) ; assert( Find(r->ID()) == r ) ; diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh index 3c9b8569..392fb622 100644 --- a/libgrive/src/drive2/Drive.hh +++ b/libgrive/src/drive2/Drive.hh @@ -57,6 +57,8 @@ namespace details typedef DB::index::type Set ; } +class Feed ; + class Drive { public : @@ -76,6 +78,7 @@ public : private : Resource* NewResource( const Json& item ) ; Resource* NewResource( http::Agent *agent, const std::string& id ) ; + void NewResource( http::Agent *agent, Feed& items ) ; private : details::DB m_db ; diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc index 40e681b5..a5832792 100644 --- a/libgrive/src/drive2/Resource.cc +++ b/libgrive/src/drive2/Resource.cc @@ -35,10 +35,15 @@ Resource::Resource() : { } -Resource::Resource( const std::string& id, const std::string& mime, const std::string& title ) : - m_id( id ), - m_mime( mime ), - m_title( title ) +Resource::Resource( + const std::string& id, + const std::string& mime, + const std::string& title, + const std::string& parent ) : + m_id ( id ), + m_mime ( mime ), + m_title ( title ), + m_parent( parent ) { } diff --git a/libgrive/src/drive2/Resource.hh b/libgrive/src/drive2/Resource.hh index 7a38275d..8513c604 100644 --- a/libgrive/src/drive2/Resource.hh +++ b/libgrive/src/drive2/Resource.hh @@ -29,7 +29,11 @@ class Resource { public : Resource() ; - Resource( const std::string& id, const std::string& mime, const std::string& title ) ; + Resource( + const std::string& id, + const std::string& mime, + const std::string& title, + const std::string& parent ) ; std::string ID() const ; std::string Mime() const ; From a668b7670b459d006f3f0f94cf597b50bda1d50d Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 23:46:49 +0800 Subject: [PATCH 045/166] added ui on the right. no logic yet --- bgrive/ui/MainWindow.ui | 60 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 7 deletions(-) diff --git a/bgrive/ui/MainWindow.ui b/bgrive/ui/MainWindow.ui index e69c6006..61c99206 100644 --- a/bgrive/ui/MainWindow.ui +++ b/bgrive/ui/MainWindow.ui @@ -28,13 +28,59 @@ - - - - 1 - 0 - - + + + + + + Title: + + + + + + + + + + + + + + + + + Filename: + + + + + + + + + + Mime type: + + + + + + + + + + Last Updated: + + + + + + + + + + From 15b3b85807e2f4ad60b79037ba088f8050455a89 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Mon, 29 Apr 2013 23:49:01 +0800 Subject: [PATCH 046/166] UT compile fixes --- libgrive/test/util/ConfigTest.hh | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/libgrive/test/util/ConfigTest.hh b/libgrive/test/util/ConfigTest.hh index f7e100d3..636ab8b8 100644 --- a/libgrive/test/util/ConfigTest.hh +++ b/libgrive/test/util/ConfigTest.hh @@ -31,15 +31,13 @@ public : ConfigTest( ) ; CPPUNIT_TEST_SUITE( ConfigTest ) ; - CPPUNIT_TEST( TestInitialiseWithEmptyString ) ; - CPPUNIT_TEST( TestInitialiseWithString ) ; - CPPUNIT_TEST( TestInitialiseWithFileSystemPath ) ; + CPPUNIT_TEST( TestInitialiseWithPath ) ; + CPPUNIT_TEST( TestInitialiseWithNoPath ) ; CPPUNIT_TEST_SUITE_END(); private : - void TestInitialiseWithEmptyString( ); - void TestInitialiseWithString( ); - void TestInitialiseWithFileSystemPath( ); + void TestInitialiseWithPath( ); + void TestInitialiseWithNoPath( ); } ; } // end of namespace From d5ca8e0afaa433696b71abd0faeb4459941c93f1 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Tue, 30 Apr 2013 00:13:13 +0800 Subject: [PATCH 047/166] show title and mime in GUI --- bgrive/src/DriveModel.cc | 1 - bgrive/src/DriveModel.hh | 1 - bgrive/src/MainWnd.cc | 25 ++++++++++++++++++++++++- bgrive/src/MainWnd.hh | 8 ++++++++ libgrive/src/drive2/Drive.cc | 6 ++++++ 5 files changed, 38 insertions(+), 3 deletions(-) diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc index e2b7d4b0..90cd2af9 100644 --- a/bgrive/src/DriveModel.cc +++ b/bgrive/src/DriveModel.cc @@ -94,7 +94,6 @@ QModelIndex DriveModel::parent( const QModelIndex& idx ) const // if my parent is root, return model index of root (i.e. QModelIndex()) const Resource *parent = m_drv.Parent(res) ; - if ( parent == 0 || parent == m_drv.Root() || idx.column() != 0 ) return QModelIndex() ; diff --git a/bgrive/src/DriveModel.hh b/bgrive/src/DriveModel.hh index 1a74283c..07d3b328 100644 --- a/bgrive/src/DriveModel.hh +++ b/bgrive/src/DriveModel.hh @@ -46,7 +46,6 @@ public : QModelIndex index( int row, int column, const QModelIndex& parent ) const ; QModelIndex parent( const QModelIndex& idx ) const ; -private : const v2::Resource* Res( const QModelIndex& idx ) const ; private : diff --git a/bgrive/src/MainWnd.cc b/bgrive/src/MainWnd.cc index 84bf64fa..28d4e088 100644 --- a/bgrive/src/MainWnd.cc +++ b/bgrive/src/MainWnd.cc @@ -20,16 +20,39 @@ #include "MainWnd.hh" +#include "drive2/Resource.hh" + +#include + #include namespace gr { +using namespace v2 ; + MainWnd::MainWnd( http::Agent *agent ) : m_drive( agent ) { m_ui.setupUi(this) ; - m_ui.m_dir->setModel( &m_drive ) ; + + connect( + m_ui.m_dir, SIGNAL(activated(const QModelIndex&)), + this, SLOT(OnClick(const QModelIndex&)) + ) ; +} + +void MainWnd::OnClick( const QModelIndex& index ) +{ + const Resource *res = m_drive.Res(index) ; + if ( res != 0 ) + ShowResource( res ) ; +} + +void MainWnd::ShowResource( const v2::Resource *res ) +{ + m_ui.m_title->setText( QString::fromUtf8(res->Title().c_str()) ) ; + m_ui.m_mime_type->setText( QString::fromUtf8(res->Mime().c_str()) ) ; } } // end of namespace diff --git a/bgrive/src/MainWnd.hh b/bgrive/src/MainWnd.hh index 5551a6b6..fe9401b6 100644 --- a/bgrive/src/MainWnd.hh +++ b/bgrive/src/MainWnd.hh @@ -25,6 +25,8 @@ #include "DriveModel.hh" +class QModelIndex ; + namespace gr { namespace http @@ -39,6 +41,12 @@ class MainWnd : public QMainWindow public : MainWnd( http::Agent *agent ) ; +private : + void ShowResource( const v2::Resource *res ) ; + +public slots : + void OnClick( const QModelIndex& index ) ; + private : Ui::MainWindow m_ui ; DriveModel m_drive ; diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index 2a28cfd0..cb3fad61 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -23,6 +23,7 @@ #include "CommonUri.hh" #include "Feed.hh" #include "protocol/Json.hh" +#include "util/Exception.hh" #include #include @@ -140,6 +141,11 @@ const Resource* Drive::Root() const const Resource* Drive::Child( const Resource *parent, std::size_t idx ) const { + if ( idx >= parent->ChildCount() ) + BOOST_THROW_EXCEPTION( + Exception() + ) ; + return Find( parent->At(idx) ) ; } From 658fb757e3cff08e5ca1b18b1eaa4d40636dc405 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Wed, 1 May 2013 01:56:34 +0800 Subject: [PATCH 048/166] added Val class for generic representation of JSON, but there's some problems with maps --- bgrive/src/DriveModel.cc | 15 ++- bgrive/src/DriveModel.hh | 5 + bgrive/src/main.cc | 29 +---- libgrive/CMakeLists.txt | 17 ++- libgrive/src/drive2/Feed.cc | 3 +- libgrive/src/drive2/Feed.hh | 3 +- libgrive/src/drive2/Val.cc | 68 ++++++++++++ libgrive/src/drive2/Val.hh | 180 ++++++++++++++++++++++++++++++++ libgrive/test/btest/UnitTest.cc | 21 ++++ libgrive/test/btest/ValTest.cc | 52 +++++++++ 10 files changed, 360 insertions(+), 33 deletions(-) create mode 100644 libgrive/src/drive2/Val.cc create mode 100644 libgrive/src/drive2/Val.hh create mode 100644 libgrive/test/btest/UnitTest.cc create mode 100644 libgrive/test/btest/ValTest.cc diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc index 90cd2af9..bc3bcf43 100644 --- a/bgrive/src/DriveModel.cc +++ b/bgrive/src/DriveModel.cc @@ -73,9 +73,20 @@ bool DriveModel::hasChildren( const QModelIndex& parent ) const return Res(parent)->ChildCount() > 0 ; } -QModelIndex DriveModel::index( int row, int column, const QModelIndex & parent ) const +QModelIndex DriveModel::index( int row, int column, const QModelIndex& parent_idx ) const { - return createIndex( row, column, const_cast(m_drv.Child(Res(parent), row)) ) ; + const Resource *parent = Res(parent_idx) ; + + // check out-of-bound + if ( parent != 0 && static_cast(row) >= parent->ChildCount() ) + BOOST_THROW_EXCEPTION( + Exception() + << InvalidRow_( row ) + << ResourceName_( parent->Title() ) + ) ; + + + return createIndex( row, column, const_cast(m_drv.Child(parent, row)) ) ; } const Resource* DriveModel::Res( const QModelIndex& idx ) const diff --git a/bgrive/src/DriveModel.hh b/bgrive/src/DriveModel.hh index 07d3b328..f39847f4 100644 --- a/bgrive/src/DriveModel.hh +++ b/bgrive/src/DriveModel.hh @@ -23,6 +23,7 @@ #include #include "drive2/Drive.hh" +#include "util/Exception.hh" namespace gr { @@ -33,6 +34,10 @@ namespace http class DriveModel : public QAbstractItemModel { +public : + typedef boost::error_info InvalidRow_ ; + typedef boost::error_info ResourceName_ ; + public : DriveModel( http::Agent *agent ) ; diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc index dd4bd939..ea578ec9 100644 --- a/bgrive/src/main.cc +++ b/bgrive/src/main.cc @@ -50,36 +50,9 @@ int main( int argc, char **argv ) Json cfg = Json::Parse( &file ) ; std::string refresh_token = cfg["refresh_token"].Str() ; - qDebug() << refresh_token.c_str() ; - OAuth2 token( refresh_token, client_id, client_secret ) ; - AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; -/* Feed feed ; - feed.Start( &agent, feed_base + "/-/folder?max-results=50&showroot=true" ) ; - - do - { - // first, get all collections from the query result - for ( Feed::iterator i = feed.begin() ; i != feed.end() ; ++i ) - { - Entry e( *i ) ; - qDebug() << e.Name().c_str() ; - } - } while ( feed.GetNext( &agent ) ) ; -*/ -/* http::JsonResponse jsp ; - agent.Get( "https://www.googleapis.com/drive/v2/files", &jsp, http::Header() ) ; - std::cout << jsp.Response() << std::endl ; -*/ -/* Feed feed( "https://www.googleapis.com/drive/v2/files" ) ; - while ( feed.Next(&agent) ) - { - std::cout << feed.Content() << std::endl ; - } -*/ -// Drive drive ; -// drive.Refresh( &agent ) ; + AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; QApplication app( argc, argv ) ; MainWnd wnd( &agent ) ; diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 08dc0660..1d429ccc 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -6,7 +6,7 @@ find_package(LibGcrypt REQUIRED) find_package(JSONC REQUIRED) find_package(CURL REQUIRED) find_package(EXPAT REQUIRED) -find_package(Boost 1.40.0 COMPONENTS program_options filesystem system REQUIRED) +find_package(Boost 1.40.0 COMPONENTS program_options filesystem unit_test_framework system REQUIRED) find_package(BFD) find_package(CppUnit) find_package(Iberty) @@ -131,3 +131,18 @@ IF ( CPPUNIT_FOUND ) ) ENDIF ( CPPUNIT_FOUND ) + +add_executable( btest + test/btest/ValTest.cc + test/btest/UnitTest.cc ) + +target_link_libraries( btest + grive + ${Boost_LIBRARIES} +) + +if ( WIN32 ) +else ( WIN32 ) + set_target_properties( btest + PROPERTIES COMPILE_FLAGS -DBOOST_TEST_DYN_LINK ) +endif (WIN32) diff --git a/libgrive/src/drive2/Feed.cc b/libgrive/src/drive2/Feed.cc index dd59ca5c..952d3a78 100644 --- a/libgrive/src/drive2/Feed.cc +++ b/libgrive/src/drive2/Feed.cc @@ -27,7 +27,8 @@ namespace gr { namespace v2 { -Feed::Feed( const std::string& base ) +Feed::Feed( const std::string& base ) : + m_base( base ) { // Next() will grab this link m_content.Add( "nextLink", Json(base) ) ; diff --git a/libgrive/src/drive2/Feed.hh b/libgrive/src/drive2/Feed.hh index 157f1e37..bbdd92ba 100644 --- a/libgrive/src/drive2/Feed.hh +++ b/libgrive/src/drive2/Feed.hh @@ -52,7 +52,8 @@ public : Json Content() const ; private : - Json m_content ; + std::string m_base ; + Json m_content ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Val.cc b/libgrive/src/drive2/Val.cc new file mode 100644 index 00000000..9c99c206 --- /dev/null +++ b/libgrive/src/drive2/Val.cc @@ -0,0 +1,68 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "Val.hh" + +namespace gr { + +Val::Val( ) : + m_base( new Impl ) +{ +} + +Val::Val( const Val& v ) : + m_base( v.m_base->Clone() ) +{ +} + +Val::~Val() +{ +} + +Val::TypeEnum Val::Type() const +{ + return m_base->Type() ; +} + +const Val& Val::operator[]( const std::string& key ) const +{ + static const Val null ; + if ( Type() == object_type ) + { + const Object& obj = As() ; + Object::const_iterator i = obj.find(key) ; + return i != obj.end() ? i->second : null ; + } + else + return null ; +} + +const Val& Val::operator[]( std::size_t index ) const +{ + static const Val null ; + return Type() == array_type ? As().at(index) : null ; +} + +void Val::Add( const std::string& key, const Val& value ) +{ + As().insert( std::make_pair(key, value) ) ; +} + +} // end of namespace diff --git a/libgrive/src/drive2/Val.hh b/libgrive/src/drive2/Val.hh new file mode 100644 index 00000000..9daa4d2a --- /dev/null +++ b/libgrive/src/drive2/Val.hh @@ -0,0 +1,180 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include +#include +#include +#include +#include + +namespace gr { + +class Val +{ +public : + enum TypeEnum { null_type, bool_type, double_type, int_type, object_type, array_type, string_type } ; + +private : + template + struct Type2Enum ; + + typedef std::vector Array ; + typedef std::map Object ; + +public : + Val() ; + Val( const Val& v ) ; + ~Val() ; + + template + Val( const T& t ) ; + + template + const T& As() const ; + + template + T& As() ; + + template + bool Is() const ; + + TypeEnum Type() const ; + + const Val& operator[]( const std::string& key ) const ; + const Val& operator[]( std::size_t index ) const ; + + void Add( const std::string& key, const Val& value ) ; + +private : + struct Base ; + + template + struct Impl ; + + std::auto_ptr m_base ; + +private : + // callback functions + static int OnNull( void *ctx ) ; + static int OnBool( void *ctx, int value ) ; + static int OnInt( void *ctx, long long value ) ; + static int OnDouble( void *ctx, double value ) ; + static int OnStr( void *ctx, const unsigned char *str, std::size_t len ) ; + static int StartMap( void *ctx ) ; + static int EndMap( void *ctx ) ; + static int StartArray( void *ctx ) ; + static int EndArray( void *ctx ) ; +} ; + +template <> +struct Val::Type2Enum +{ + static const TypeEnum type = null_type ; +} ; + +template <> +struct Val::Type2Enum +{ + static const TypeEnum type = int_type ; +} ; + +template <> +struct Val::Type2Enum +{ + static const TypeEnum type = bool_type ; +} ; + +template <> +struct Val::Type2Enum +{ + static const TypeEnum type = double_type ; +} ; + +template <> +struct Val::Type2Enum +{ + static const TypeEnum type = string_type ; +} ; + +template <> +struct Val::Type2Enum +{ + static const TypeEnum type = array_type ; +} ; + +template <> +struct Val::Type2Enum +{ + static const TypeEnum type = object_type ; +} ; + +struct Val::Base +{ + virtual ~Base() {} + virtual Base* Clone() const = 0 ; + virtual TypeEnum Type() const = 0 ; +} ; + +template +struct Val::Impl : public Base +{ + T val ; + Impl( const T& t ) : val(t) {} + Impl* Clone() const { return new Impl(val); } + TypeEnum Type() const { return Type2Enum::type ; } +} ; + +template <> +struct Val::Impl : public Base +{ + Impl* Clone() const { return new Impl; } + TypeEnum Type() const { return null_type ; } +} ; + +template +Val::Val( const T& t ) : + m_base( new Impl(t) ) +{ +} + +template +const T& Val::As() const +{ + const Impl *impl = &dynamic_cast&>( *m_base ) ; + return impl->val ; +} + +template +T& Val::As() +{ + Impl *impl = &dynamic_cast&>( *m_base ) ; + return impl->val ; +} + +template +bool Val::Is() const +{ + return Type() == Type2Enum::type ; +} + +} // end of namespace + diff --git a/libgrive/test/btest/UnitTest.cc b/libgrive/test/btest/UnitTest.cc new file mode 100644 index 00000000..5a824cff --- /dev/null +++ b/libgrive/test/btest/UnitTest.cc @@ -0,0 +1,21 @@ +/* + webwrite: an GPL wiki-like website with in-place editing + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#define BOOST_TEST_MAIN +#include diff --git a/libgrive/test/btest/ValTest.cc b/libgrive/test/btest/ValTest.cc new file mode 100644 index 00000000..aecb0a13 --- /dev/null +++ b/libgrive/test/btest/ValTest.cc @@ -0,0 +1,52 @@ +/* + webwrite: an GPL wiki-like website with in-place editing + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "drive2/Val.hh" +#include + +using namespace gr ; + +namespace +{ + struct Fixture + { + } ; +} + +BOOST_FIXTURE_TEST_SUITE( ValTest, Fixture ) + +BOOST_AUTO_TEST_CASE( TestSimpleTypes ) +{ + Val null ; + BOOST_CHECK_EQUAL( null.Type(), Val::null_type ) ; + BOOST_CHECK( null.Is() ) ; + + Val i( 100 ) ; + BOOST_CHECK_EQUAL( i.As(), 100 ) ; + BOOST_CHECK_EQUAL( i.Type(), Val::int_type ) ; +} + +BOOST_AUTO_TEST_CASE( TestMap ) +{ + Val obj ; + obj.Add( "key", Val( std::string("value") ) ) ; + BOOST_CHECK_EQUAL( obj["key"].As(), "value" ) ; +} + +BOOST_AUTO_TEST_SUITE_END() From 540e3d82dbb395451f6df5b9248bc68485b773fe Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Wed, 1 May 2013 13:36:29 +0800 Subject: [PATCH 049/166] added type conversion to supported types --- libgrive/src/drive2/Val.cc | 46 ++++++++--- libgrive/src/drive2/Val.hh | 137 +++++++++++++++++++++------------ libgrive/test/btest/ValTest.cc | 5 +- 3 files changed, 126 insertions(+), 62 deletions(-) diff --git a/libgrive/src/drive2/Val.cc b/libgrive/src/drive2/Val.cc index 9c99c206..a36512b6 100644 --- a/libgrive/src/drive2/Val.cc +++ b/libgrive/src/drive2/Val.cc @@ -20,6 +20,8 @@ #include "Val.hh" +#include + namespace gr { Val::Val( ) : @@ -43,21 +45,25 @@ Val::TypeEnum Val::Type() const const Val& Val::operator[]( const std::string& key ) const { - static const Val null ; - if ( Type() == object_type ) - { - const Object& obj = As() ; - Object::const_iterator i = obj.find(key) ; - return i != obj.end() ? i->second : null ; - } - else - return null ; + const Object& obj = As() ; + Object::const_iterator i = obj.find(key) ; + if ( i != obj.end() ) + return i->second ; + + // shut off compiler warning + BOOST_THROW_EXCEPTION(Error() << NoKey_(key)) ; + throw ; } const Val& Val::operator[]( std::size_t index ) const { - static const Val null ; - return Type() == array_type ? As().at(index) : null ; + const Array& ar = As() ; + if ( index < ar.size() ) + return ar[index] ; + + // shut off compiler warning + BOOST_THROW_EXCEPTION(Error() << OutOfRange_(index)) ; + throw ; } void Val::Add( const std::string& key, const Val& value ) @@ -65,4 +71,22 @@ void Val::Add( const std::string& key, const Val& value ) As().insert( std::make_pair(key, value) ) ; } +void Val::Swap( Val& val ) +{ + std::swap( m_base, val.m_base ) ; +} + } // end of namespace + +namespace std +{ + void swap( gr::Val& v1, gr::Val& v2 ) + { + v1.Swap( v2 ) ; + } + + ostream& operator<<( ostream& os, gr::Val::TypeEnum t ) + { + return os << static_cast(t) ; + } +} diff --git a/libgrive/src/drive2/Val.hh b/libgrive/src/drive2/Val.hh index 9daa4d2a..1fae9d58 100644 --- a/libgrive/src/drive2/Val.hh +++ b/libgrive/src/drive2/Val.hh @@ -20,11 +20,14 @@ #pragma once +#include "util/Exception.hh" + #include #include #include #include #include +#include namespace gr { @@ -33,10 +36,20 @@ class Val public : enum TypeEnum { null_type, bool_type, double_type, int_type, object_type, array_type, string_type } ; + struct Error : virtual Exception {} ; + typedef boost::error_info SrcType_ ; + typedef boost::error_info DestType_ ; + typedef boost::error_info NoKey_ ; + typedef boost::error_info OutOfRange_ ; + private : template struct Type2Enum ; + template + struct SupportType ; + +public : typedef std::vector Array ; typedef std::map Object ; @@ -46,8 +59,28 @@ public : ~Val() ; template - Val( const T& t ) ; + explicit Val( const T& t ) + { + Assign(t) ; + } + + template + Val& Assign( const T& t ) ; + void Swap( Val& val ) ; + Val& operator=( const Val& val ) + { + Val tmp(val) ; + Swap(tmp) ; + return *this ; + } + + template + Val& operator=( const T& t ) + { + return Assign(t) ; + } + template const T& As() const ; @@ -85,47 +118,27 @@ private : static int EndArray( void *ctx ) ; } ; -template <> -struct Val::Type2Enum -{ - static const TypeEnum type = null_type ; -} ; - -template <> -struct Val::Type2Enum -{ - static const TypeEnum type = int_type ; -} ; - -template <> -struct Val::Type2Enum -{ - static const TypeEnum type = bool_type ; -} ; - -template <> -struct Val::Type2Enum -{ - static const TypeEnum type = double_type ; -} ; - -template <> -struct Val::Type2Enum -{ - static const TypeEnum type = string_type ; -} ; - -template <> -struct Val::Type2Enum -{ - static const TypeEnum type = array_type ; -} ; - -template <> -struct Val::Type2Enum -{ - static const TypeEnum type = object_type ; -} ; +template <> struct Val::Type2Enum { static const TypeEnum type = null_type ; } ; +template <> struct Val::Type2Enum { static const TypeEnum type = int_type ; } ; +template <> struct Val::Type2Enum { static const TypeEnum type = bool_type ; } ; +template <> struct Val::Type2Enum { static const TypeEnum type = double_type ;} ; +template <> struct Val::Type2Enum { static const TypeEnum type = string_type ; } ; +template <> struct Val::Type2Enum { static const TypeEnum type = array_type ; } ; +template <> struct Val::Type2Enum { static const TypeEnum type = object_type ; } ; + +template <> struct Val::SupportType { typedef long long Type ; } ; +template <> struct Val::SupportType { typedef long long Type ; } ; +template <> struct Val::SupportType { typedef long long Type ; } ; +template <> struct Val::SupportType { typedef long long Type ; } ; +template <> struct Val::SupportType { typedef long long Type ; } ; +template <> struct Val::SupportType { typedef long long Type ; } ; + +template <> struct Val::SupportType { typedef bool Type ; } ; +template <> struct Val::SupportType { typedef double Type ; } ; +template <> struct Val::SupportType { typedef std::string Type ; } ; +template <> struct Val::SupportType { typedef std::string Type ; } ; +template <> struct Val::SupportType { typedef Val::Array Type ; } ; +template <> struct Val::SupportType { typedef Val::Object Type ; } ; struct Val::Base { @@ -135,7 +148,7 @@ struct Val::Base } ; template -struct Val::Impl : public Base +struct Val::Impl : public Base { T val ; Impl( const T& t ) : val(t) {} @@ -151,23 +164,44 @@ struct Val::Impl : public Base } ; template -Val::Val( const T& t ) : - m_base( new Impl(t) ) +Val& Val::Assign( const T& t ) { + m_base.reset( new Impl::Type>(t) ) ; + return *this ; } template const T& Val::As() const { - const Impl *impl = &dynamic_cast&>( *m_base ) ; - return impl->val ; + try + { + const Impl *impl = &dynamic_cast&>( *m_base ) ; + return impl->val ; + } + catch ( std::exception& e ) + { + TypeEnum dest = Type2Enum::type ; + BOOST_THROW_EXCEPTION( + Error() << SrcType_(Type()) << DestType_(dest) + ) ; + } } template T& Val::As() { - Impl *impl = &dynamic_cast&>( *m_base ) ; - return impl->val ; + try + { + Impl *impl = &dynamic_cast&>( *m_base ) ; + return impl->val ; + } + catch ( std::exception& e ) + { + TypeEnum dest = Type2Enum::type ; + BOOST_THROW_EXCEPTION( + Error() << SrcType_(Type()) << DestType_(dest) + ) ; + } } template @@ -178,3 +212,8 @@ bool Val::Is() const } // end of namespace +namespace std +{ + void swap( gr::Val& v1, gr::Val& v2 ) ; + ostream& operator<<( ostream& os, gr::Val::TypeEnum t ) ; +} diff --git a/libgrive/test/btest/ValTest.cc b/libgrive/test/btest/ValTest.cc index aecb0a13..cf4d49f4 100644 --- a/libgrive/test/btest/ValTest.cc +++ b/libgrive/test/btest/ValTest.cc @@ -38,14 +38,15 @@ BOOST_AUTO_TEST_CASE( TestSimpleTypes ) BOOST_CHECK( null.Is() ) ; Val i( 100 ) ; - BOOST_CHECK_EQUAL( i.As(), 100 ) ; + BOOST_CHECK_EQUAL( i.As(), 100 ) ; BOOST_CHECK_EQUAL( i.Type(), Val::int_type ) ; } BOOST_AUTO_TEST_CASE( TestMap ) { - Val obj ; + Val obj(( Val::Object() )) ; obj.Add( "key", Val( std::string("value") ) ) ; + BOOST_CHECK_EQUAL( obj["key"].Type(), Val::string_type ) ; BOOST_CHECK_EQUAL( obj["key"].As(), "value" ) ; } From 2d29692601c653591632b256c5822fa39a0ca9c0 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Wed, 1 May 2013 22:47:37 +0800 Subject: [PATCH 050/166] added yajl parser --- libgrive/CMakeLists.txt | 9 +- libgrive/src/drive2/JsonVal.cc | 135 +++++++++++++++++++++++++++++ libgrive/src/drive2/JsonVal.hh | 41 +++++++++ libgrive/src/drive2/Val.hh | 12 +-- libgrive/src/drive2/ValBuilder.cc | 130 +++++++++++++++++++++++++++ libgrive/src/drive2/ValBuilder.hh | 67 ++++++++++++++ libgrive/test/btest/JsonValTest.cc | 43 +++++++++ 7 files changed, 424 insertions(+), 13 deletions(-) create mode 100644 libgrive/src/drive2/JsonVal.cc create mode 100644 libgrive/src/drive2/JsonVal.hh create mode 100644 libgrive/src/drive2/ValBuilder.cc create mode 100644 libgrive/src/drive2/ValBuilder.hh create mode 100644 libgrive/test/btest/JsonValTest.cc diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 1d429ccc..6f2ba0d0 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -79,6 +79,7 @@ add_definitions( add_library( grive STATIC ${LIBGRIVE_SRC} ${OPT_SRC} ) target_link_libraries( grive + yajl ${CURL_LIBRARIES} ${JSONC_LIBRARY} ${LIBGCRYPT_LIBRARIES} @@ -132,9 +133,11 @@ IF ( CPPUNIT_FOUND ) ENDIF ( CPPUNIT_FOUND ) -add_executable( btest - test/btest/ValTest.cc - test/btest/UnitTest.cc ) +file(GLOB BTEST_SRC + test/btest/*.cc +) + +add_executable( btest ${BTEST_SRC} ) target_link_libraries( btest grive diff --git a/libgrive/src/drive2/JsonVal.cc b/libgrive/src/drive2/JsonVal.cc new file mode 100644 index 00000000..5f333f55 --- /dev/null +++ b/libgrive/src/drive2/JsonVal.cc @@ -0,0 +1,135 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "JsonVal.hh" + +#include "Val.hh" +#include "ValBuilder.hh" + +#include + +namespace gr { namespace json { + +namespace +{ + int OnNull( void *ctx ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->BuildNull() ; + return true ; + } + + int OnBool( void *ctx, int value ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->Build( static_cast(value) ) ; + return true ; + } + + int OnInt( void *ctx, long long value ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->Build(value) ; + return true ; + } + + int OnDouble( void *ctx, double value ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->Build(value) ; + return true ; + } + + int OnStr( void *ctx, const unsigned char *str, std::size_t len ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->Build( std::string(reinterpret_cast(str), len) ) ; + return true ; + } + + int StartMap( void *ctx ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->StartObject() ; + return true ; + } + + int OnMapKey( void *ctx, const unsigned char *str, std::size_t len ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->BuildKey( std::string(reinterpret_cast(str), len) ) ; + return true ; + } + + int EndMap( void *ctx ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->EndObject() ; + return true ; + } + + int StartArray( void *ctx ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->StartArray() ; + return true ; + } + + int EndArray( void *ctx ) + { + ValBuilder *b = reinterpret_cast(ctx) ; + b->EndArray() ; + return true ; + } + + const yajl_callbacks callbacks = { + OnNull, + OnBool, + OnInt, + OnDouble, + 0, + OnStr, + StartMap, + OnMapKey, + EndMap, + StartArray, + EndArray, + }; +} + +Val Parse( const std::string& json ) +{ + ValBuilder b ; + yajl_handle hand = yajl_alloc( &callbacks, 0, &b ) ; + yajl_parse( hand, reinterpret_cast(json.c_str()), json.size() ) ; + + if ( yajl_complete_parse(hand) != yajl_status_ok ) + { + unsigned char *msg = yajl_get_error( hand, true, reinterpret_cast(json.c_str()), json.size() ) ; + std::string msg_str(reinterpret_cast(msg)) ; + yajl_free_error(hand, msg) ; + + BOOST_THROW_EXCEPTION( Error() << ParseErr_(msg_str) << JsonText_(json) ) ; + } + + return b.Result() ; +} + +} } // end of namespace gr::json diff --git a/libgrive/src/drive2/JsonVal.hh b/libgrive/src/drive2/JsonVal.hh new file mode 100644 index 00000000..1831d71f --- /dev/null +++ b/libgrive/src/drive2/JsonVal.hh @@ -0,0 +1,41 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include "util/Exception.hh" + +#include + +namespace gr +{ + class Val ; + + namespace json + { + struct Error : virtual Exception {} ; + typedef boost::error_info ParseErr_ ; + typedef boost::error_info JsonText_ ; + + Val Parse( const std::string& json ) ; + } + +} // end of namespace + diff --git a/libgrive/src/drive2/Val.hh b/libgrive/src/drive2/Val.hh index 1fae9d58..4f542934 100644 --- a/libgrive/src/drive2/Val.hh +++ b/libgrive/src/drive2/Val.hh @@ -106,16 +106,6 @@ private : std::auto_ptr m_base ; private : - // callback functions - static int OnNull( void *ctx ) ; - static int OnBool( void *ctx, int value ) ; - static int OnInt( void *ctx, long long value ) ; - static int OnDouble( void *ctx, double value ) ; - static int OnStr( void *ctx, const unsigned char *str, std::size_t len ) ; - static int StartMap( void *ctx ) ; - static int EndMap( void *ctx ) ; - static int StartArray( void *ctx ) ; - static int EndArray( void *ctx ) ; } ; template <> struct Val::Type2Enum { static const TypeEnum type = null_type ; } ; @@ -132,6 +122,8 @@ template <> struct Val::SupportType { typedef long long Type ; } ; template <> struct Val::SupportType { typedef long long Type ; } ; template <> struct Val::SupportType { typedef long long Type ; } ; template <> struct Val::SupportType { typedef long long Type ; } ; +template <> struct Val::SupportType { typedef long long Type ; } ; +template <> struct Val::SupportType { typedef long long Type ; } ; template <> struct Val::SupportType { typedef bool Type ; } ; template <> struct Val::SupportType { typedef double Type ; } ; diff --git a/libgrive/src/drive2/ValBuilder.cc b/libgrive/src/drive2/ValBuilder.cc new file mode 100644 index 00000000..e11f454b --- /dev/null +++ b/libgrive/src/drive2/ValBuilder.cc @@ -0,0 +1,130 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "ValBuilder.hh" + +#include + +namespace gr { + +ValBuilder::ValBuilder( ) +{ +} + +ValBuilder::~ValBuilder() +{ +} + +void ValBuilder::Build( long long t ) +{ + Build(Val(t)) ; +} + +void ValBuilder::Build( double t ) +{ + Build(Val(t)) ; +} + +void ValBuilder::Build( const std::string& t ) +{ + Build(Val(t)) ; +} + +void ValBuilder::Build( bool t ) +{ + Build(Val(t)) ; +} + +void ValBuilder::BuildNull() +{ + Build(Val()) ; +} + +void ValBuilder::Build( const Val& t ) +{ + if ( m_ctx.empty() ) + m_ctx.push( t ) ; + + else if ( m_ctx.top().Is() ) + { + Val::Array& ar = m_ctx.top().As() ; + ar.push_back( t ) ; + } + else if ( m_ctx.top().Is() ) + { + if ( m_key.get() == 0 ) + BOOST_THROW_EXCEPTION( Error() << NoKey_(t) ) ; + + else + { + Val::Object& obj = m_ctx.top().As() ; + obj.insert( std::make_pair( m_key->As(), t ) ) ; + m_key.reset() ; + } + } + else + BOOST_THROW_EXCEPTION( Error() << Unexpected_(m_ctx.top()) ) ; +} + +void ValBuilder::BuildKey( const std::string& t ) +{ + m_key.reset( new Val(t) ) ; +} + +void ValBuilder::StartArray() +{ + m_ctx.push( Val( Val::Array() ) ) ; +} + +void ValBuilder::EndArray() +{ + End( Val::array_type ) ; +} + +void ValBuilder::End( Val::TypeEnum type ) +{ + if ( m_ctx.top().Type() == type ) + { + // get top Val from stack + Val current ; + current.Swap( m_ctx.top() ) ; + m_ctx.pop() ; + + Build(current) ; + } +} + +void ValBuilder::StartObject() +{ + m_ctx.push( Val( Val::Object() ) ) ; +} + +void ValBuilder::EndObject() +{ + End( Val::object_type ) ; +} + +Val ValBuilder::Result() const +{ + assert( m_ctx.size() == 1U ) ; + return m_ctx.top() ; +} + +} // end of namespace diff --git a/libgrive/src/drive2/ValBuilder.hh b/libgrive/src/drive2/ValBuilder.hh new file mode 100644 index 00000000..96bf610a --- /dev/null +++ b/libgrive/src/drive2/ValBuilder.hh @@ -0,0 +1,67 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include "Val.hh" +#include "util/Exception.hh" + +#include +#include + +namespace gr { + +class ValBuilder +{ +public : + struct Error : virtual Exception {} ; + typedef boost::error_info Mismatch_ ; + typedef boost::error_info Unexpected_ ; + typedef boost::error_info NoKey_ ; + +public : + ValBuilder( ) ; + ~ValBuilder() ; + + void Build( long long t ) ; + void Build( double t ) ; + void Build( const std::string& t ) ; + void Build( bool t ) ; + void BuildNull() ; + void Build( const Val& t ) ; + + void StartArray() ; + void EndArray() ; + void StartObject() ; + void BuildKey( const std::string& t ) ; + void EndObject() ; + + Val Result() const ; + +private : + void End( Val::TypeEnum type ) ; + +private : + std::stack m_ctx ; + std::auto_ptr m_key ; +} ; + +} // end of namespace + diff --git a/libgrive/test/btest/JsonValTest.cc b/libgrive/test/btest/JsonValTest.cc new file mode 100644 index 00000000..1772d87a --- /dev/null +++ b/libgrive/test/btest/JsonValTest.cc @@ -0,0 +1,43 @@ +/* + webwrite: an GPL wiki-like website with in-place editing + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "drive2/JsonVal.hh" +#include "drive2/Val.hh" + +#include + +using namespace gr ; + +namespace +{ + struct F + { + } ; +} + +BOOST_FIXTURE_TEST_SUITE( JsonValTest, F ) + +BOOST_AUTO_TEST_CASE( Test ) +{ + Val json = json::Parse( "{\"key\": 100 }" ) ; + BOOST_CHECK( json.Is() ) ; + BOOST_CHECK_EQUAL( json["key"].As(), 100 ) ; +} + +BOOST_AUTO_TEST_SUITE_END() From abfa9ce76524ced1236eb5c5b1d97947840b4ff9 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 2 May 2013 00:04:42 +0800 Subject: [PATCH 051/166] added ValResponse class to parse JSON from http --- libgrive/CMakeLists.txt | 1 + libgrive/src/drive2/JsonVal.cc | 135 -------------- libgrive/src/json/JsonParser.cc | 174 ++++++++++++++++++ libgrive/src/json/JsonParser.hh | 53 ++++++ libgrive/src/{drive2 => json}/Val.cc | 64 ++++++- libgrive/src/{drive2 => json}/Val.hh | 39 ++-- libgrive/src/{drive2 => json}/ValBuilder.cc | 12 +- libgrive/src/{drive2 => json}/ValBuilder.hh | 16 +- libgrive/src/json/ValResponse.cc | 53 ++++++ .../JsonVal.hh => json/ValResponse.hh} | 33 ++-- libgrive/src/json/ValVisitor.hh | 46 +++++ libgrive/test/btest/JsonValTest.cc | 10 +- libgrive/test/btest/ValTest.cc | 2 +- 13 files changed, 460 insertions(+), 178 deletions(-) delete mode 100644 libgrive/src/drive2/JsonVal.cc create mode 100644 libgrive/src/json/JsonParser.cc create mode 100644 libgrive/src/json/JsonParser.hh rename libgrive/src/{drive2 => json}/Val.cc (69%) rename libgrive/src/{drive2 => json}/Val.hh (86%) rename libgrive/src/{drive2 => json}/ValBuilder.cc (90%) rename libgrive/src/{drive2 => json}/ValBuilder.hh (85%) create mode 100644 libgrive/src/json/ValResponse.cc rename libgrive/src/{drive2/JsonVal.hh => json/ValResponse.hh} (65%) create mode 100644 libgrive/src/json/ValVisitor.hh diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 6f2ba0d0..3cc15742 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -65,6 +65,7 @@ file (GLOB LIBGRIVE_SRC src/drive2/*.cc src/http/*.cc src/protocol/*.cc + src/json/*.cc src/util/*.cc src/util/log/*.cc src/xml/*.cc diff --git a/libgrive/src/drive2/JsonVal.cc b/libgrive/src/drive2/JsonVal.cc deleted file mode 100644 index 5f333f55..00000000 --- a/libgrive/src/drive2/JsonVal.cc +++ /dev/null @@ -1,135 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#include "JsonVal.hh" - -#include "Val.hh" -#include "ValBuilder.hh" - -#include - -namespace gr { namespace json { - -namespace -{ - int OnNull( void *ctx ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->BuildNull() ; - return true ; - } - - int OnBool( void *ctx, int value ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->Build( static_cast(value) ) ; - return true ; - } - - int OnInt( void *ctx, long long value ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->Build(value) ; - return true ; - } - - int OnDouble( void *ctx, double value ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->Build(value) ; - return true ; - } - - int OnStr( void *ctx, const unsigned char *str, std::size_t len ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->Build( std::string(reinterpret_cast(str), len) ) ; - return true ; - } - - int StartMap( void *ctx ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->StartObject() ; - return true ; - } - - int OnMapKey( void *ctx, const unsigned char *str, std::size_t len ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->BuildKey( std::string(reinterpret_cast(str), len) ) ; - return true ; - } - - int EndMap( void *ctx ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->EndObject() ; - return true ; - } - - int StartArray( void *ctx ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->StartArray() ; - return true ; - } - - int EndArray( void *ctx ) - { - ValBuilder *b = reinterpret_cast(ctx) ; - b->EndArray() ; - return true ; - } - - const yajl_callbacks callbacks = { - OnNull, - OnBool, - OnInt, - OnDouble, - 0, - OnStr, - StartMap, - OnMapKey, - EndMap, - StartArray, - EndArray, - }; -} - -Val Parse( const std::string& json ) -{ - ValBuilder b ; - yajl_handle hand = yajl_alloc( &callbacks, 0, &b ) ; - yajl_parse( hand, reinterpret_cast(json.c_str()), json.size() ) ; - - if ( yajl_complete_parse(hand) != yajl_status_ok ) - { - unsigned char *msg = yajl_get_error( hand, true, reinterpret_cast(json.c_str()), json.size() ) ; - std::string msg_str(reinterpret_cast(msg)) ; - yajl_free_error(hand, msg) ; - - BOOST_THROW_EXCEPTION( Error() << ParseErr_(msg_str) << JsonText_(json) ) ; - } - - return b.Result() ; -} - -} } // end of namespace gr::json diff --git a/libgrive/src/json/JsonParser.cc b/libgrive/src/json/JsonParser.cc new file mode 100644 index 00000000..9ae63f48 --- /dev/null +++ b/libgrive/src/json/JsonParser.cc @@ -0,0 +1,174 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "JsonParser.hh" + +#include "Val.hh" +#include "ValBuilder.hh" + +#include + +namespace gr { + +namespace +{ + int OnNull( void *ctx ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->VisitNull() ; + return true ; + } + + int OnBool( void *ctx, int value ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->Visit( static_cast(value) ) ; + return true ; + } + + int OnInt( void *ctx, long long value ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->Visit(value) ; + return true ; + } + + int OnDouble( void *ctx, double value ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->Visit(value) ; + return true ; + } + + int OnStr( void *ctx, const unsigned char *str, std::size_t len ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->Visit( std::string(reinterpret_cast(str), len) ) ; + return true ; + } + + int StartMap( void *ctx ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->StartObject() ; + return true ; + } + + int OnMapKey( void *ctx, const unsigned char *str, std::size_t len ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->VisitKey( std::string(reinterpret_cast(str), len) ) ; + return true ; + } + + int EndMap( void *ctx ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->EndObject() ; + return true ; + } + + int StartArray( void *ctx ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->StartArray() ; + return true ; + } + + int EndArray( void *ctx ) + { + ValVisitor *b = reinterpret_cast(ctx) ; + b->EndArray() ; + return true ; + } + + const yajl_callbacks callbacks = { + OnNull, + OnBool, + OnInt, + OnDouble, + 0, + OnStr, + StartMap, + OnMapKey, + EndMap, + StartArray, + EndArray, + }; +} + +void JsonParser::Parse( const std::string& json, ValVisitor *callback ) +{ + JsonParser parser( callback ) ; + parser.Parse( json.c_str(), json.size() ) ; + parser.Finish() ; +} + +struct JsonParser::Impl +{ + ValVisitor *callback ; + yajl_handle hand ; +} ; + +JsonParser::JsonParser( ValVisitor *callback ) : + m_impl( new Impl ) +{ + m_impl->callback = callback ; + m_impl->hand = yajl_alloc( &callbacks, 0, m_impl->callback ) ; +} + +JsonParser::~JsonParser() +{ + yajl_free( m_impl->hand ) ; +} + +void JsonParser::Parse( const char *str, std::size_t size ) +{ + const unsigned char *ustr = reinterpret_cast(str) ; + + yajl_status r = yajl_parse( m_impl->hand, ustr, size ) ; + + if ( r != yajl_status_ok ) + { + unsigned char *msg = yajl_get_error( m_impl->hand, true, ustr, size ) ; + std::string msg_str(reinterpret_cast(msg)) ; + yajl_free_error(m_impl->hand, msg) ; + + BOOST_THROW_EXCEPTION( + Error() + << ParseErr_(msg_str) + << JsonText_(std::string(str,size)) + ); + } +} + +void JsonParser::Finish() +{ + if ( yajl_complete_parse(m_impl->hand) != yajl_status_ok ) + { + unsigned char *msg = yajl_get_error( m_impl->hand, false, 0, 0 ) ; + std::string msg_str(reinterpret_cast(msg)) ; + yajl_free_error(m_impl->hand, msg) ; + + BOOST_THROW_EXCEPTION( Error() << ParseErr_(msg_str) ) ; + } +} + +} // end of namespace gr::json diff --git a/libgrive/src/json/JsonParser.hh b/libgrive/src/json/JsonParser.hh new file mode 100644 index 00000000..27d6a4c8 --- /dev/null +++ b/libgrive/src/json/JsonParser.hh @@ -0,0 +1,53 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include "util/Exception.hh" + +#include +#include + +namespace gr { + +class ValVisitor ; + +class JsonParser +{ +public : + struct Error : virtual Exception {} ; + typedef boost::error_info ParseErr_ ; + typedef boost::error_info JsonText_ ; + + static void Parse( const std::string& json, ValVisitor *callback ) ; + + explicit JsonParser( ValVisitor *callback ) ; + ~JsonParser() ; + + void Parse( const char *str, std::size_t size ) ; + void Finish() ; + +private : + struct Impl ; + std::auto_ptr m_impl ; +} ; + +} // end of namespace + diff --git a/libgrive/src/drive2/Val.cc b/libgrive/src/json/Val.cc similarity index 69% rename from libgrive/src/drive2/Val.cc rename to libgrive/src/json/Val.cc index a36512b6..38bdc13d 100644 --- a/libgrive/src/drive2/Val.cc +++ b/libgrive/src/json/Val.cc @@ -38,6 +38,18 @@ Val::~Val() { } +void Val::Swap( Val& val ) +{ + std::swap( m_base, val.m_base ) ; +} + +Val& Val::operator=( const Val& val ) +{ + Val tmp(val) ; + Swap(tmp) ; + return *this ; +} + Val::TypeEnum Val::Type() const { return m_base->Type() ; @@ -66,14 +78,58 @@ const Val& Val::operator[]( std::size_t index ) const throw ; } -void Val::Add( const std::string& key, const Val& value ) +std::string Val::Str() const { - As().insert( std::make_pair(key, value) ) ; + return As() ; } -void Val::Swap( Val& val ) +int Val::Int() const { - std::swap( m_base, val.m_base ) ; + return static_cast(As()) ; +} + +double Val::Double() const +{ + return As() ; +} + +bool Val::Bool() const +{ + return As() ; +} + +const Val::Array& Val::AsArray() const +{ + return As() ; +} + +const Val::Object& Val::AsObject() const +{ + return As() ; +} + +bool Val::Has( const std::string& key ) const +{ + const Object& obj = As() ; + return obj.find(key) != obj.end() ; +} + +bool Val::Get( const std::string& key, Val& val ) const +{ + const Object& obj = As() ; + Object::const_iterator i = obj.find(key) ; + if ( i != obj.end() ) + { + val = i->second ; + return true ; + } + else + return false ; +} + +void Val::Add( const std::string& key, const Val& value ) +{ + As().insert( std::make_pair(key, value) ) ; } } // end of namespace diff --git a/libgrive/src/drive2/Val.hh b/libgrive/src/json/Val.hh similarity index 86% rename from libgrive/src/drive2/Val.hh rename to libgrive/src/json/Val.hh index 4f542934..7c3bf317 100644 --- a/libgrive/src/drive2/Val.hh +++ b/libgrive/src/json/Val.hh @@ -23,14 +23,16 @@ #include "util/Exception.hh" #include -#include -#include +#include #include +#include #include -#include +#include namespace gr { +class ValVisitor ; + class Val { public : @@ -66,14 +68,9 @@ public : template Val& Assign( const T& t ) ; - void Swap( Val& val ) ; - Val& operator=( const Val& val ) - { - Val tmp(val) ; - Swap(tmp) ; - return *this ; - } + void Swap( Val& val ) ; + Val& operator=( const Val& val ) ; template Val& operator=( const T& t ) @@ -95,7 +92,27 @@ public : const Val& operator[]( const std::string& key ) const ; const Val& operator[]( std::size_t index ) const ; - void Add( const std::string& key, const Val& value ) ; + // shortcuts for As<>() + std::string Str() const ; + int Int() const ; + long Long() const ; + double Double() const ; + bool Bool() const ; + const Array& AsArray() const ; + const Object& AsObject() const ; + + // shortcuts for objects + bool Has( const std::string& key ) const ; + bool Get( const std::string& key, Val& val ) const ; + void Add( const std::string& key, const Val& val ) ; + + // shortcuts for array (and array of objects) + void Add( const Val& json ) ; + Val FindInArray( const std::string& key, const std::string& value ) const ; + bool FindInArray( const std::string& key, const std::string& value, Val& result ) const ; + +// friend std::ostream& operator<<( std::ostream& os, const Val& json ) ; +// void Visit( DataStream *out ) const ; private : struct Base ; diff --git a/libgrive/src/drive2/ValBuilder.cc b/libgrive/src/json/ValBuilder.cc similarity index 90% rename from libgrive/src/drive2/ValBuilder.cc rename to libgrive/src/json/ValBuilder.cc index e11f454b..a80be2d1 100644 --- a/libgrive/src/drive2/ValBuilder.cc +++ b/libgrive/src/json/ValBuilder.cc @@ -32,27 +32,27 @@ ValBuilder::~ValBuilder() { } -void ValBuilder::Build( long long t ) +void ValBuilder::Visit( long long t ) { Build(Val(t)) ; } -void ValBuilder::Build( double t ) +void ValBuilder::Visit( double t ) { Build(Val(t)) ; } -void ValBuilder::Build( const std::string& t ) +void ValBuilder::Visit( const std::string& t ) { Build(Val(t)) ; } -void ValBuilder::Build( bool t ) +void ValBuilder::Visit( bool t ) { Build(Val(t)) ; } -void ValBuilder::BuildNull() +void ValBuilder::VisitNull() { Build(Val()) ; } @@ -83,7 +83,7 @@ void ValBuilder::Build( const Val& t ) BOOST_THROW_EXCEPTION( Error() << Unexpected_(m_ctx.top()) ) ; } -void ValBuilder::BuildKey( const std::string& t ) +void ValBuilder::VisitKey( const std::string& t ) { m_key.reset( new Val(t) ) ; } diff --git a/libgrive/src/drive2/ValBuilder.hh b/libgrive/src/json/ValBuilder.hh similarity index 85% rename from libgrive/src/drive2/ValBuilder.hh rename to libgrive/src/json/ValBuilder.hh index 96bf610a..f5c2ab3e 100644 --- a/libgrive/src/drive2/ValBuilder.hh +++ b/libgrive/src/json/ValBuilder.hh @@ -20,6 +20,8 @@ #pragma once +#include "ValVisitor.hh" + #include "Val.hh" #include "util/Exception.hh" @@ -28,7 +30,7 @@ namespace gr { -class ValBuilder +class ValBuilder : public ValVisitor { public : struct Error : virtual Exception {} ; @@ -40,17 +42,17 @@ public : ValBuilder( ) ; ~ValBuilder() ; - void Build( long long t ) ; - void Build( double t ) ; - void Build( const std::string& t ) ; - void Build( bool t ) ; - void BuildNull() ; + void Visit( long long t ) ; + void Visit( double t ) ; + void Visit( const std::string& t ) ; + void Visit( bool t ) ; + void VisitNull() ; void Build( const Val& t ) ; void StartArray() ; void EndArray() ; void StartObject() ; - void BuildKey( const std::string& t ) ; + void VisitKey( const std::string& t ) ; void EndObject() ; Val Result() const ; diff --git a/libgrive/src/json/ValResponse.cc b/libgrive/src/json/ValResponse.cc new file mode 100644 index 00000000..02c81b48 --- /dev/null +++ b/libgrive/src/json/ValResponse.cc @@ -0,0 +1,53 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "ValResponse.hh" + +#include "Val.hh" + +namespace gr { namespace http { + +ValResponse::ValResponse( ) : + m_parser( &m_val ) +{ +} + +std::size_t ValResponse::Write( const char *data, std::size_t count ) +{ + m_parser.Parse( data, count ) ; + return count ; +} + +std::size_t ValResponse::Read( char *data, std::size_t count ) +{ + return count ; +} + +Val ValResponse::Response() const +{ + return m_val.Result() ; +} + +void ValResponse::Finish() +{ + m_parser.Finish() ; +} + +} } // end of namespace gr::http diff --git a/libgrive/src/drive2/JsonVal.hh b/libgrive/src/json/ValResponse.hh similarity index 65% rename from libgrive/src/drive2/JsonVal.hh rename to libgrive/src/json/ValResponse.hh index 1831d71f..cdeca6af 100644 --- a/libgrive/src/drive2/JsonVal.hh +++ b/libgrive/src/json/ValResponse.hh @@ -20,22 +20,33 @@ #pragma once -#include "util/Exception.hh" +#include "util/DataStream.hh" -#include +#include "JsonParser.hh" +#include "ValBuilder.hh" namespace gr { class Val ; +} - namespace json - { - struct Error : virtual Exception {} ; - typedef boost::error_info ParseErr_ ; - typedef boost::error_info JsonText_ ; - - Val Parse( const std::string& json ) ; - } +namespace gr { namespace http { + +class ValResponse : public DataStream +{ +public : + ValResponse() ; + + std::size_t Write( const char *data, std::size_t count ) ; + std::size_t Read( char *data, std::size_t count ) ; + + void Finish() ; + Val Response() const ; -} // end of namespace +private : + ValBuilder m_val ; + JsonParser m_parser ; +} ; + +} } // end of namespace gr::http diff --git a/libgrive/src/json/ValVisitor.hh b/libgrive/src/json/ValVisitor.hh new file mode 100644 index 00000000..b523c70d --- /dev/null +++ b/libgrive/src/json/ValVisitor.hh @@ -0,0 +1,46 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include + +namespace gr { + +class ValVisitor +{ +public : + virtual ~ValVisitor() {} + + virtual void Visit( long long t ) = 0 ; + virtual void Visit( double t ) = 0 ; + virtual void Visit( const std::string& t ) = 0 ; + virtual void Visit( bool t ) = 0 ; + virtual void VisitNull() = 0 ; + + virtual void StartArray() = 0 ; + virtual void EndArray() = 0 ; + virtual void StartObject() = 0 ; + virtual void VisitKey( const std::string& t ) = 0 ; + virtual void EndObject() = 0 ; +} ; + +} // end of namespace + diff --git a/libgrive/test/btest/JsonValTest.cc b/libgrive/test/btest/JsonValTest.cc index 1772d87a..bb8fc2a4 100644 --- a/libgrive/test/btest/JsonValTest.cc +++ b/libgrive/test/btest/JsonValTest.cc @@ -17,8 +17,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "drive2/JsonVal.hh" -#include "drive2/Val.hh" +#include "json/JsonParser.hh" +#include "json/Val.hh" +#include "json/ValBuilder.hh" #include @@ -35,7 +36,10 @@ BOOST_FIXTURE_TEST_SUITE( JsonValTest, F ) BOOST_AUTO_TEST_CASE( Test ) { - Val json = json::Parse( "{\"key\": 100 }" ) ; + ValBuilder b ; + JsonParser::Parse( "{\"key\": 100 }", &b ) ; + Val json = b.Result() ; + BOOST_CHECK( json.Is() ) ; BOOST_CHECK_EQUAL( json["key"].As(), 100 ) ; } diff --git a/libgrive/test/btest/ValTest.cc b/libgrive/test/btest/ValTest.cc index cf4d49f4..5514825e 100644 --- a/libgrive/test/btest/ValTest.cc +++ b/libgrive/test/btest/ValTest.cc @@ -17,7 +17,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "drive2/Val.hh" +#include "json/Val.hh" #include using namespace gr ; From 6fa6e690c6d27ad21bc008ed966616485269dd11 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 2 May 2013 00:41:47 +0800 Subject: [PATCH 052/166] added json writer --- libgrive/src/json/Val.cc | 40 +++++++++++ libgrive/src/json/Val.hh | 2 +- libgrive/src/json/ValWriter.cc | 111 +++++++++++++++++++++++++++++ libgrive/src/json/ValWriter.hh | 57 +++++++++++++++ libgrive/src/util/StringStream.cc | 67 +++++++++++++++++ libgrive/src/util/StringStream.hh | 56 +++++++++++++++ libgrive/test/btest/JsonValTest.cc | 10 +++ 7 files changed, 342 insertions(+), 1 deletion(-) create mode 100644 libgrive/src/json/ValWriter.cc create mode 100644 libgrive/src/json/ValWriter.hh create mode 100644 libgrive/src/util/StringStream.cc create mode 100644 libgrive/src/util/StringStream.hh diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index 38bdc13d..e4c2a855 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -19,6 +19,7 @@ */ #include "Val.hh" +#include "ValVisitor.hh" #include @@ -132,6 +133,45 @@ void Val::Add( const std::string& key, const Val& value ) As().insert( std::make_pair(key, value) ) ; } +void Val::Visit( ValVisitor *visitor ) const +{ + switch ( Type() ) + { + case null_type: visitor->VisitNull() ; break ; + case int_type: visitor->Visit( As() ) ; break ; + case double_type: visitor->Visit( As() ) ; break ; + case string_type: visitor->Visit( As() ) ; break ; + case bool_type: visitor->Visit( As() ) ; break ; + + case object_type: + { + visitor->StartObject() ; + + const Object& obj = As() ; + for ( Object::const_iterator i = obj.begin() ; i != obj.end() ; ++i ) + { + visitor->VisitKey( i->first ) ; + i->second.Visit( visitor ) ; + } + + visitor->EndObject() ; + break ; + } + + case array_type: + { + visitor->StartArray() ; + + const Array& arr = As() ; + for ( Array::const_iterator i = arr.begin() ; i != arr.end() ; ++i ) + i->Visit( visitor ) ; + + visitor->EndArray() ; + break ; + } + } +} + } // end of namespace namespace std diff --git a/libgrive/src/json/Val.hh b/libgrive/src/json/Val.hh index 7c3bf317..46003f05 100644 --- a/libgrive/src/json/Val.hh +++ b/libgrive/src/json/Val.hh @@ -112,7 +112,7 @@ public : bool FindInArray( const std::string& key, const std::string& value, Val& result ) const ; // friend std::ostream& operator<<( std::ostream& os, const Val& json ) ; -// void Visit( DataStream *out ) const ; + void Visit( ValVisitor *visitor ) const ; private : struct Base ; diff --git a/libgrive/src/json/ValWriter.cc b/libgrive/src/json/ValWriter.cc new file mode 100644 index 00000000..502e4325 --- /dev/null +++ b/libgrive/src/json/ValWriter.cc @@ -0,0 +1,111 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "ValWriter.hh" +#include "util/DataStream.hh" + +#include + +#include + +namespace gr { + +struct ValWriter::Impl +{ + yajl_gen gen ; + DataStream *out ; +} ; + +ValWriter::ValWriter( DataStream *out ) : + m_impl( new Impl ) +{ + assert( out != 0 ) ; + + m_impl->out = out ; + m_impl->gen = yajl_gen_alloc(0) ; + yajl_gen_config( m_impl->gen, yajl_gen_print_callback, &ValWriter::WriteCallback, this ) ; +} + +ValWriter::~ValWriter() +{ + yajl_gen_free( m_impl->gen ) ; +} + +void ValWriter::Visit( long long t ) +{ + yajl_gen_integer( m_impl->gen, t ) ; +} + +void ValWriter::Visit( double t ) +{ + yajl_gen_double( m_impl->gen, t ) ; +} + +void ValWriter::Visit( const std::string& t ) +{ + yajl_gen_string( m_impl->gen, + reinterpret_cast(t.c_str()), t.size() ) ; +} + +void ValWriter::Visit( bool t ) +{ + yajl_gen_bool( m_impl->gen, t ) ; +} + +void ValWriter::VisitNull() +{ + yajl_gen_null( m_impl->gen ) ; +} + +void ValWriter::StartArray() +{ + yajl_gen_array_open( m_impl->gen ) ; +} + +void ValWriter::EndArray() +{ + yajl_gen_array_close( m_impl->gen ) ; +} + +void ValWriter::StartObject() +{ + yajl_gen_map_open( m_impl->gen ) ; +} + +void ValWriter::VisitKey( const std::string& t ) +{ + Visit(t) ; +} + +void ValWriter::EndObject() +{ + yajl_gen_map_close( m_impl->gen ) ; +} + +void ValWriter::WriteCallback( void *ctx, const char *str, std::size_t size ) +{ + ValWriter *pthis = reinterpret_cast(ctx) ; + assert( pthis != 0 ) ; + assert( pthis->m_impl->out != 0 ) ; + + pthis->m_impl->out->Write( str, size ) ; +} + +} // end of namespace diff --git a/libgrive/src/json/ValWriter.hh b/libgrive/src/json/ValWriter.hh new file mode 100644 index 00000000..351cbd85 --- /dev/null +++ b/libgrive/src/json/ValWriter.hh @@ -0,0 +1,57 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include "ValVisitor.hh" +#include + +namespace gr { + +class DataStream ; + +class ValWriter : public ValVisitor +{ +public : + ValWriter( DataStream *out ) ; + ~ValWriter() ; + + void Visit( long long t ) ; + void Visit( double t ) ; + void Visit( const std::string& t ) ; + void Visit( bool t ) ; + void VisitNull() ; + + void StartArray() ; + void EndArray() ; + void StartObject() ; + void VisitKey( const std::string& t ) ; + void EndObject() ; + +private : + static void WriteCallback( void *ctx, const char *str, std::size_t size ) ; + +private : + struct Impl ; + std::auto_ptr m_impl ; +} ; + +} // end of namespace + diff --git a/libgrive/src/util/StringStream.cc b/libgrive/src/util/StringStream.cc new file mode 100644 index 00000000..2cefdfed --- /dev/null +++ b/libgrive/src/util/StringStream.cc @@ -0,0 +1,67 @@ +/* + webwrite: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "StringStream.hh" + +#include + +namespace gr { + +namespace +{ + // the max size of the cached string. this is to prevent allocating + // too much memory if client sends a line too long (i.e. DOS attack) + const std::size_t capacity = 1024 ; +} + +StringStream::StringStream( const std::string& init ) : + m_str( init ) +{ +} + +/// Read `size` bytes from the stream. Those bytes will be removed from +/// the underlying string by calling `std::string::erase()`. Therefore, it is +/// not a good idea to call Read() to read byte-by-byte. +std::size_t StringStream::Read( char *data, std::size_t size ) +{ + // wow! no need to count count == 0 + std::size_t count = std::min( m_str.size(), size ) ; + std::copy( m_str.begin(), m_str.begin() + count, data ) ; + m_str.erase( 0, count ) ; + return count ; +} + +std::size_t StringStream::Write( const char *data, std::size_t size ) +{ + std::size_t count = std::min( size, capacity - m_str.size() ) ; + m_str.insert( m_str.end(), data, data+count ) ; + return count ; +} + +const std::string& StringStream::Str() const +{ + return m_str ; +} + +void StringStream::Str( const std::string& str ) +{ + m_str = str ; +} + +} // end of namespace diff --git a/libgrive/src/util/StringStream.hh b/libgrive/src/util/StringStream.hh new file mode 100644 index 00000000..600f24c3 --- /dev/null +++ b/libgrive/src/util/StringStream.hh @@ -0,0 +1,56 @@ +/* + webwrite: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "DataStream.hh" + +#include + +namespace gr { + +/** \brief DataStream base on `std::string`s + + StringStream is a DataStream back-end that uses std::string for storage. + It is useful for unit tests and text parsing, especially when used with + StreamParser. + + StringStream has a limit on the maximum number of byte it stores. This + is to prevent DOS attacks in which the client sends a lot of bytes before + the delimiter (e.g. new-line characters) and the server is forced to hold + all of them in memory. + + The limit is current 1024 bytes. +*/ +class StringStream : public DataStream +{ +public : + explicit StringStream( const std::string& init = std::string() ) ; + + std::size_t Read( char *data, std::size_t size ) ; + std::size_t Write( const char *data, std::size_t size ) ; + + const std::string& Str() const ; + void Str( const std::string& str ) ; + +private : + std::string m_str ; +} ; + +} // end of namespace diff --git a/libgrive/test/btest/JsonValTest.cc b/libgrive/test/btest/JsonValTest.cc index bb8fc2a4..61e98dac 100644 --- a/libgrive/test/btest/JsonValTest.cc +++ b/libgrive/test/btest/JsonValTest.cc @@ -20,6 +20,8 @@ #include "json/JsonParser.hh" #include "json/Val.hh" #include "json/ValBuilder.hh" +#include "json/ValWriter.hh" +#include "util/StringStream.hh" #include @@ -42,6 +44,14 @@ BOOST_AUTO_TEST_CASE( Test ) BOOST_CHECK( json.Is() ) ; BOOST_CHECK_EQUAL( json["key"].As(), 100 ) ; + + StringStream ss ; + ValWriter wr( &ss ) ; + json.Visit( &wr ) ; + + BOOST_CHECK_EQUAL( ss.Str(), "{\"key\":100}" ) ; + +// std::cout << ss.Str() << std::endl ; } BOOST_AUTO_TEST_SUITE_END() From 6ba04dcfba110bca134fc72f055d2be991771a1e Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Thu, 2 May 2013 00:55:39 +0800 Subject: [PATCH 053/166] added operator<< to print out json --- .../src/json/{ValWriter.cc => JsonWriter.cc} | 34 +++++++-------- .../src/json/{ValWriter.hh => JsonWriter.hh} | 6 +-- libgrive/src/json/Val.cc | 11 +++++ libgrive/src/json/Val.hh | 2 +- libgrive/src/util/StdStream.cc | 42 +++++++++++++++++++ libgrive/src/util/StdStream.hh | 42 +++++++++++++++++++ libgrive/test/btest/JsonValTest.cc | 4 +- 7 files changed, 118 insertions(+), 23 deletions(-) rename libgrive/src/json/{ValWriter.cc => JsonWriter.cc} (69%) rename libgrive/src/json/{ValWriter.hh => JsonWriter.hh} (93%) create mode 100644 libgrive/src/util/StdStream.cc create mode 100644 libgrive/src/util/StdStream.hh diff --git a/libgrive/src/json/ValWriter.cc b/libgrive/src/json/JsonWriter.cc similarity index 69% rename from libgrive/src/json/ValWriter.cc rename to libgrive/src/json/JsonWriter.cc index 502e4325..9fbcc2bb 100644 --- a/libgrive/src/json/ValWriter.cc +++ b/libgrive/src/json/JsonWriter.cc @@ -18,7 +18,7 @@ MA 02110-1301, USA. */ -#include "ValWriter.hh" +#include "JsonWriter.hh" #include "util/DataStream.hh" #include @@ -27,81 +27,81 @@ namespace gr { -struct ValWriter::Impl +struct JsonWriter::Impl { yajl_gen gen ; DataStream *out ; } ; -ValWriter::ValWriter( DataStream *out ) : +JsonWriter::JsonWriter( DataStream *out ) : m_impl( new Impl ) { assert( out != 0 ) ; m_impl->out = out ; m_impl->gen = yajl_gen_alloc(0) ; - yajl_gen_config( m_impl->gen, yajl_gen_print_callback, &ValWriter::WriteCallback, this ) ; + yajl_gen_config( m_impl->gen, yajl_gen_print_callback, &JsonWriter::WriteCallback, this ) ; } -ValWriter::~ValWriter() +JsonWriter::~JsonWriter() { yajl_gen_free( m_impl->gen ) ; } -void ValWriter::Visit( long long t ) +void JsonWriter::Visit( long long t ) { yajl_gen_integer( m_impl->gen, t ) ; } -void ValWriter::Visit( double t ) +void JsonWriter::Visit( double t ) { yajl_gen_double( m_impl->gen, t ) ; } -void ValWriter::Visit( const std::string& t ) +void JsonWriter::Visit( const std::string& t ) { yajl_gen_string( m_impl->gen, reinterpret_cast(t.c_str()), t.size() ) ; } -void ValWriter::Visit( bool t ) +void JsonWriter::Visit( bool t ) { yajl_gen_bool( m_impl->gen, t ) ; } -void ValWriter::VisitNull() +void JsonWriter::VisitNull() { yajl_gen_null( m_impl->gen ) ; } -void ValWriter::StartArray() +void JsonWriter::StartArray() { yajl_gen_array_open( m_impl->gen ) ; } -void ValWriter::EndArray() +void JsonWriter::EndArray() { yajl_gen_array_close( m_impl->gen ) ; } -void ValWriter::StartObject() +void JsonWriter::StartObject() { yajl_gen_map_open( m_impl->gen ) ; } -void ValWriter::VisitKey( const std::string& t ) +void JsonWriter::VisitKey( const std::string& t ) { Visit(t) ; } -void ValWriter::EndObject() +void JsonWriter::EndObject() { yajl_gen_map_close( m_impl->gen ) ; } -void ValWriter::WriteCallback( void *ctx, const char *str, std::size_t size ) +void JsonWriter::WriteCallback( void *ctx, const char *str, std::size_t size ) { - ValWriter *pthis = reinterpret_cast(ctx) ; + JsonWriter *pthis = reinterpret_cast(ctx) ; assert( pthis != 0 ) ; assert( pthis->m_impl->out != 0 ) ; diff --git a/libgrive/src/json/ValWriter.hh b/libgrive/src/json/JsonWriter.hh similarity index 93% rename from libgrive/src/json/ValWriter.hh rename to libgrive/src/json/JsonWriter.hh index 351cbd85..350357d2 100644 --- a/libgrive/src/json/ValWriter.hh +++ b/libgrive/src/json/JsonWriter.hh @@ -27,11 +27,11 @@ namespace gr { class DataStream ; -class ValWriter : public ValVisitor +class JsonWriter : public ValVisitor { public : - ValWriter( DataStream *out ) ; - ~ValWriter() ; + JsonWriter( DataStream *out ) ; + ~JsonWriter() ; void Visit( long long t ) ; void Visit( double t ) ; diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index e4c2a855..02f7dffa 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -19,7 +19,9 @@ */ #include "Val.hh" +#include "JsonWriter.hh" #include "ValVisitor.hh" +#include "util/StdStream.hh" #include @@ -172,6 +174,15 @@ void Val::Visit( ValVisitor *visitor ) const } } +std::ostream& operator<<( std::ostream& os, const Val& val ) +{ + StdStream ss( os.rdbuf() ) ; + JsonWriter wr( &ss ) ; + val.Visit( &wr ) ; + + return os ; +} + } // end of namespace namespace std diff --git a/libgrive/src/json/Val.hh b/libgrive/src/json/Val.hh index 46003f05..56e43bcc 100644 --- a/libgrive/src/json/Val.hh +++ b/libgrive/src/json/Val.hh @@ -111,7 +111,7 @@ public : Val FindInArray( const std::string& key, const std::string& value ) const ; bool FindInArray( const std::string& key, const std::string& value, Val& result ) const ; -// friend std::ostream& operator<<( std::ostream& os, const Val& json ) ; + friend std::ostream& operator<<( std::ostream& os, const Val& val ) ; void Visit( ValVisitor *visitor ) const ; private : diff --git a/libgrive/src/util/StdStream.cc b/libgrive/src/util/StdStream.cc new file mode 100644 index 00000000..eaa10cea --- /dev/null +++ b/libgrive/src/util/StdStream.cc @@ -0,0 +1,42 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#include "StdStream.hh" + +#include + +namespace gr { + +StdStream::StdStream( std::streambuf *buf ) : + m_adaptee( buf ) +{ +} + +std::size_t StdStream::Read( char *data, std::size_t size ) +{ + return m_adaptee == 0 ? 0 : m_adaptee->sgetn( data, size ) ; +} + +std::size_t StdStream::Write( const char *data, std::size_t size ) +{ + return m_adaptee == 0 ? 0 : m_adaptee->sputn( data, size ) ; +} + +} // end of namespace diff --git a/libgrive/src/util/StdStream.hh b/libgrive/src/util/StdStream.hh new file mode 100644 index 00000000..4751e388 --- /dev/null +++ b/libgrive/src/util/StdStream.hh @@ -0,0 +1,42 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2013 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, + MA 02110-1301, USA. +*/ + +#pragma once + +#include "DataStream.hh" + +#include + +namespace gr { + +class StdStream : public DataStream +{ +public : + explicit StdStream( std::streambuf *buf ) ; + + std::size_t Read( char *data, std::size_t size ) ; + std::size_t Write( const char *data, std::size_t size ) ; + +private : + std::streambuf *m_adaptee ; +} ; + +} // end of namespace + diff --git a/libgrive/test/btest/JsonValTest.cc b/libgrive/test/btest/JsonValTest.cc index 61e98dac..d5087cdf 100644 --- a/libgrive/test/btest/JsonValTest.cc +++ b/libgrive/test/btest/JsonValTest.cc @@ -20,7 +20,7 @@ #include "json/JsonParser.hh" #include "json/Val.hh" #include "json/ValBuilder.hh" -#include "json/ValWriter.hh" +#include "json/JsonWriter.hh" #include "util/StringStream.hh" #include @@ -46,7 +46,7 @@ BOOST_AUTO_TEST_CASE( Test ) BOOST_CHECK_EQUAL( json["key"].As(), 100 ) ; StringStream ss ; - ValWriter wr( &ss ) ; + JsonWriter wr( &ss ) ; json.Visit( &wr ) ; BOOST_CHECK_EQUAL( ss.Str(), "{\"key\":100}" ) ; From 27817e835fe115ebbda5410ec904aa49a2ad01f1 Mon Sep 17 00:00:00 2001 From: Nestal Wan Date: Fri, 3 May 2013 00:40:04 +0800 Subject: [PATCH 054/166] using the new Json classes for bgrive --- libgrive/src/drive2/Drive.cc | 18 ++++++++---- libgrive/src/drive2/Drive.hh | 4 +-- libgrive/src/drive2/Feed.cc | 12 ++++---- libgrive/src/drive2/Feed.hh | 10 +++---- libgrive/src/json/Val.cc | 52 ++++++++++++++++++++++++++++++++- libgrive/src/json/Val.hh | 11 +++++-- libgrive/src/json/ValBuilder.cc | 37 +++++++++++++---------- libgrive/src/json/ValBuilder.hh | 9 ++++-- 8 files changed, 114 insertions(+), 39 deletions(-) diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc index cb3fad61..feaa84be 100644 --- a/libgrive/src/drive2/Drive.cc +++ b/libgrive/src/drive2/Drive.cc @@ -22,9 +22,12 @@ #include "CommonUri.hh" #include "Feed.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" #include "util/Exception.hh" +#include + +#include #include #include @@ -74,8 +77,8 @@ void Drive::NewResource( http::Agent *agent, Feed& items ) while ( items.Next( agent ) ) { - std::vector item_json = items.Content()["items"].AsArray() ; - for ( std::vector::iterator i = item_json.begin() ; i != item_json.end() ; ++i ) + std::vector item_json = items.Content()["items"].AsArray() ; + for ( std::vector::iterator i = item_json.begin() ; i != item_json.end() ; ++i ) NewResource( *i ) ; } } @@ -89,16 +92,19 @@ Resource* Drive::NewResource( http::Agent *agent, const std::string& id ) return NewResource( feed.Content() ) ; } -Resource* Drive::NewResource( const Json& item ) +Resource* Drive::NewResource( const Val& item ) { // assume resource is directly under root std::string parent_id = m_root != 0 ? m_root->ID() : "" ; - Json parents ; + Val parents ; if ( item.Get( "parents", parents ) ) { + std::vector pids_val = parents.Select( "id" ) ; std::vector pids ; - parents.Select( "id", std::back_inserter(pids) ) ; + std::transform( pids_val.begin(), pids_val.end(), + std::back_inserter( pids ), + boost::bind( &Val::Str, _1 ) ) ; // only the first parent counts if ( !pids.empty() ) diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh index 392fb622..33014b03 100644 --- a/libgrive/src/drive2/Drive.hh +++ b/libgrive/src/drive2/Drive.hh @@ -34,7 +34,7 @@ namespace http class Agent ; } -class Json ; +class Val ; namespace v2 { @@ -76,7 +76,7 @@ public : const Resource* Parent( const Resource *child ) const ; private : - Resource* NewResource( const Json& item ) ; + Resource* NewResource( const Val& item ) ; Resource* NewResource( http::Agent *agent, const std::string& id ) ; void NewResource( http::Agent *agent, Feed& items ) ; diff --git a/libgrive/src/drive2/Feed.cc b/libgrive/src/drive2/Feed.cc index 952d3a78..78f67f06 100644 --- a/libgrive/src/drive2/Feed.cc +++ b/libgrive/src/drive2/Feed.cc @@ -21,7 +21,7 @@ #include "http/Agent.hh" #include "http/Header.hh" -#include "protocol/JsonResponse.hh" +#include "json/ValResponse.hh" #include @@ -31,22 +31,22 @@ Feed::Feed( const std::string& base ) : m_base( base ) { // Next() will grab this link - m_content.Add( "nextLink", Json(base) ) ; + m_content.Add( "nextLink", Val(base) ) ; } void Feed::Query( const std::string& field, const std::string& value ) { std::string url = m_content["nextLink"].Str() ; - m_content.Add( "nextLink", Json( url + "?q=" + field + "+%3d+%27" + value + "%27" ) ) ; + m_content.Add( "nextLink", Val( url + "?q=" + field + "+%3d+%27" + value + "%27" ) ) ; } bool Feed::Next( http::Agent *agent ) { - Json url ; + Val url ; if ( !m_content.Get("nextLink", url) ) return false ; - http::JsonResponse out ; + http::ValResponse out ; try { agent->Get( url.Str(), &out, http::Header() ) ; @@ -61,7 +61,7 @@ bool Feed::Next( http::Agent *agent ) return true ; } -Json Feed::Content() const +Val Feed::Content() const { return m_content ; } diff --git a/libgrive/src/drive2/Feed.hh b/libgrive/src/drive2/Feed.hh index bbdd92ba..ad1f294a 100644 --- a/libgrive/src/drive2/Feed.hh +++ b/libgrive/src/drive2/Feed.hh @@ -19,7 +19,7 @@ #pragma once -#include "protocol/Json.hh" +#include "json/Val.hh" #include "util/Exception.hh" #include @@ -32,7 +32,7 @@ namespace http class Header ; } -class Json ; +class Val ; namespace v2 { @@ -40,7 +40,7 @@ class Feed { public : // exception info - typedef boost::error_info DriveFeed_ ; + typedef boost::error_info DriveFeed_ ; public : Feed( const std::string& base ) ; @@ -49,11 +49,11 @@ public : bool Next( http::Agent *agent ) ; - Json Content() const ; + Val Content() const ; private : std::string m_base ; - Json m_content ; + Val m_content ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index 02f7dffa..79c0b5e1 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -27,11 +27,32 @@ namespace gr { +const Val& Val::Null() +{ + static const Val null( null_type ) ; + return null ; +} + Val::Val( ) : - m_base( new Impl ) + m_base( new Impl ) { } +Val::Val( TypeEnum type ) +{ + switch ( type ) + { + case int_type: m_base.reset( new Impl ) ; break ; + case bool_type: m_base.reset( new Impl ) ; break ; + case double_type: m_base.reset( new Impl ) ; break ; + case string_type: m_base.reset( new Impl ) ; break ; + case array_type: m_base.reset( new Impl ) ; break ; + case object_type: m_base.reset( new Impl ) ; break ; + case null_type: + default: m_base.reset( new Impl ) ; break ; + } +} + Val::Val( const Val& v ) : m_base( v.m_base->Clone() ) { @@ -174,6 +195,35 @@ void Val::Visit( ValVisitor *visitor ) const } } +void Val::Select( const Object& obj, const std::string& key, std::vector& result ) const +{ + Object::const_iterator i = obj.find(key) ; + if ( i != obj.end() ) + result.push_back(i->second) ; +} + +/** If \a this is an array of objects, this function returns all values of + the objects in the array with the key \a key. If \a this is an object, + just return the value with the key \a key. +*/ +std::vector Val::Select( const std::string& key ) const +{ + std::vector result ; + if ( Is() ) + Select( As(), key, result ) ; + + else if ( Is() ) + { + const Array& array = As() ; + for ( Array::const_iterator i = array.begin() ; i != array.end() ; ++i ) + { + if ( i->Is() ) + Select( i->As(), key, result ) ; + } + } + return result ; +} + std::ostream& operator<<( std::ostream& os, const Val& val ) { StdStream ss( os.rdbuf() ) ; diff --git a/libgrive/src/json/Val.hh b/libgrive/src/json/Val.hh index 56e43bcc..25478860 100644 --- a/libgrive/src/json/Val.hh +++ b/libgrive/src/json/Val.hh @@ -58,8 +58,11 @@ public : public : Val() ; Val( const Val& v ) ; + explicit Val( TypeEnum type ) ; ~Val() ; + static const Val& Null() ; + template explicit Val( const T& t ) { @@ -110,19 +113,22 @@ public : void Add( const Val& json ) ; Val FindInArray( const std::string& key, const std::string& value ) const ; bool FindInArray( const std::string& key, const std::string& value, Val& result ) const ; - + + std::vector Select( const std::string& key ) const ; + friend std::ostream& operator<<( std::ostream& os, const Val& val ) ; void Visit( ValVisitor *visitor ) const ; private : struct Base ; - + template struct Impl ; std::auto_ptr m_base ; private : + void Select( const Object& obj, const std::string& key, std::vector& result ) const ; } ; template <> struct Val::Type2Enum { static const TypeEnum type = null_type ; } ; @@ -160,6 +166,7 @@ template struct Val::Impl : public Base { T val ; + Impl( ) : val() {} Impl( const T& t ) : val(t) {} Impl* Clone() const { return new Impl(val); } TypeEnum Type() const { return Type2Enum::type ; } diff --git a/libgrive/src/json/ValBuilder.cc b/libgrive/src/json/ValBuilder.cc index a80be2d1..c5658ff2 100644 --- a/libgrive/src/json/ValBuilder.cc +++ b/libgrive/src/json/ValBuilder.cc @@ -60,37 +60,41 @@ void ValBuilder::VisitNull() void ValBuilder::Build( const Val& t ) { if ( m_ctx.empty() ) - m_ctx.push( t ) ; + { + Level l = { Val::Null(), t } ; + m_ctx.push( l ) ; + } - else if ( m_ctx.top().Is() ) + else if ( m_ctx.top().val.Is() ) { - Val::Array& ar = m_ctx.top().As() ; + Val::Array& ar = m_ctx.top().val.As() ; ar.push_back( t ) ; } - else if ( m_ctx.top().Is() ) + else if ( m_ctx.top().val.Is() ) { - if ( m_key.get() == 0 ) + if ( !m_ctx.top().key.Is() ) BOOST_THROW_EXCEPTION( Error() << NoKey_(t) ) ; else { - Val::Object& obj = m_ctx.top().As() ; - obj.insert( std::make_pair( m_key->As(), t ) ) ; - m_key.reset() ; + Val::Object& obj = m_ctx.top().val.As() ; + obj.insert( std::make_pair( m_ctx.top().key.Str(), t ) ) ; + m_ctx.top().key = Val::Null() ; } } else - BOOST_THROW_EXCEPTION( Error() << Unexpected_(m_ctx.top()) ) ; + BOOST_THROW_EXCEPTION( Error() << Unexpected_(m_ctx.top().val) ) ; } void ValBuilder::VisitKey( const std::string& t ) { - m_key.reset( new Val(t) ) ; + m_ctx.top().key = t ; } void ValBuilder::StartArray() { - m_ctx.push( Val( Val::Array() ) ) ; + Level l = { Val::Null(), Val(Val::Array()) } ; + m_ctx.push(l) ; } void ValBuilder::EndArray() @@ -100,11 +104,13 @@ void ValBuilder::EndArray() void ValBuilder::End( Val::TypeEnum type ) { - if ( m_ctx.top().Type() == type ) + if ( m_ctx.top().val.Type() == type ) { + assert( m_ctx.top().key.Is() ) ; + // get top Val from stack Val current ; - current.Swap( m_ctx.top() ) ; + current.Swap( m_ctx.top().val ) ; m_ctx.pop() ; Build(current) ; @@ -113,7 +119,8 @@ void ValBuilder::End( Val::TypeEnum type ) void ValBuilder::StartObject() { - m_ctx.push( Val( Val::Object() ) ) ; + Level l = { Val::Null(), Val( Val::Object() ) } ; + m_ctx.push(l) ; } void ValBuilder::EndObject() @@ -124,7 +131,7 @@ void ValBuilder::EndObject() Val ValBuilder::Result() const { assert( m_ctx.size() == 1U ) ; - return m_ctx.top() ; + return m_ctx.top().val ; } } // end of namespace diff --git a/libgrive/src/json/ValBuilder.hh b/libgrive/src/json/ValBuilder.hh index f5c2ab3e..efbe5b2a 100644 --- a/libgrive/src/json/ValBuilder.hh +++ b/libgrive/src/json/ValBuilder.hh @@ -61,8 +61,13 @@ private : void End( Val::TypeEnum type ) ; private : - std::stack m_ctx ; - std::auto_ptr m_key ; + struct Level + { + Val key ; + Val val ; + } ; + + std::stack m_ctx ; } ; } // end of namespace From 752eb3fddaac72f26047ffb46e3e76c4a13908bf Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 8 Sep 2013 02:36:30 +0400 Subject: [PATCH 055/166] Allow to sync just one directory in the root --- grive/src/main.cc | 1 + libgrive/src/drive/State.cc | 9 +++++++++ libgrive/src/drive/State.hh | 1 + libgrive/src/util/Config.cc | 3 +++ 4 files changed, 14 insertions(+) diff --git a/grive/src/main.cc b/grive/src/main.cc index 5209c941..b7b25852 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -110,6 +110,7 @@ int Main( int argc, char **argv ) ( "version,v", "Display Grive version" ) ( "auth,a", "Request authorization token" ) ( "path,p", po::value(), "Path to sync") + ( "dir,s", po::value(), "Subdirectory to sync") ( "verbose,V", "Verbose mode. Enable more messages than normal.") ( "log-xml", "Log more HTTP responses as XML for debugging.") ( "new-rev", "Create new revisions in server for updated files.") diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 2d1dc943..cf0c30bd 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -35,6 +35,7 @@ namespace gr { namespace v1 { State::State( const fs::path& filename, const Json& options ) : m_res ( options["path"].Str() ), + m_dir ( options["dir"].Str() ), m_cstamp ( -1 ) { Read( filename ) ; @@ -78,6 +79,10 @@ void State::FromLocal( const fs::path& p, Resource* folder ) if ( IsIgnore(fname) ) Log( "file %1% is ignored by grive", fname, log::verbose ) ; + // check if it is ignored + else if ( folder == m_res.Root() && m_dir != "" && fname != m_dir ) + Log( "%1% %2% is ignored", fs::is_directory(i->path()) ? "folder" : "file", fname, log::verbose ); + // check for broken symblic links else if ( !fs::exists( i->path() ) ) Log( "file %1% doesn't exist (broken link?), ignored", i->path(), log::verbose ) ; @@ -109,6 +114,10 @@ void State::FromRemote( const Entry& e ) if ( IsIgnore( e.Name() ) ) Log( "%1% %2% is ignored by grive", e.Kind(), e.Name(), log::verbose ) ; + // check if it is ignored + else if ( e.ParentHref() == m_res.Root()->SelfHref() && m_dir != "" && e.Name() != m_dir ) + Log( "%1% %2% is ignored", e.Kind(), e.Name(), log::verbose ); + // common checkings else if ( e.Kind() != "folder" && (fn.empty() || e.ContentSrc().empty()) ) Log( "%1% \"%2%\" is a google document, ignored", e.Kind(), e.Name(), log::verbose ) ; diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index 8c641892..98cce39a 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -79,6 +79,7 @@ private : ResourceTree m_res ; DateTime m_last_sync ; long m_cstamp ; + std::string m_dir ; std::vector m_unresolved ; } ; diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index b5a9f15b..12c22c76 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -42,6 +42,9 @@ Config::Config( const po::variables_map& vm ) m_cmd.Add( "path", Json(vm.count("path") > 0 ? vm["path"].as() : default_root_folder ) ) ; + m_cmd.Add( "dir", Json(vm.count("dir") > 0 + ? vm["dir"].as() + : "" ) ) ; m_path = GetPath( fs::path(m_cmd["path"].Str()) ) ; m_file = Read( ) ; From 89506b4162935e529b90f16afa059e36a27c18f0 Mon Sep 17 00:00:00 2001 From: Visa Putkinen Date: Sun, 24 Nov 2013 14:22:49 +0200 Subject: [PATCH 056/166] Add autogenerated stuff to .gitignore --- .gitignore | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.gitignore b/.gitignore index da71e17a..6fa3768d 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,13 @@ grive.kdev4 .project .cproject build/ +/CMakeCache.txt +CMakeFiles +moc_*.cxx* +bgrive/ui_MainWindow.h +Makefile +*.a +bgrive/bgrive +grive/grive +libgrive/btest +*.cmake From c15bfbe8bc93e4d59810a0c47225e98fd8269428 Mon Sep 17 00:00:00 2001 From: Visa Putkinen Date: Sun, 24 Nov 2013 14:25:01 +0200 Subject: [PATCH 057/166] Ignore only .grive and .grive_state --- libgrive/src/drive/State.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 2d1dc943..124adecc 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -60,7 +60,7 @@ void State::FromLocal( const fs::path& p ) bool State::IsIgnore( const std::string& filename ) { - return filename[0] == '.' ; + return filename == ".grive" || filename == ".grive_state"; } void State::FromLocal( const fs::path& p, Resource* folder ) From 8b434dad144341cc2b49d95a7ab4b048ca2b3a0c Mon Sep 17 00:00:00 2001 From: Visa Putkinen Date: Sun, 24 Nov 2013 14:34:03 +0200 Subject: [PATCH 058/166] Reduce number of syscalls while scanning local files --- libgrive/src/drive/Resource.cc | 6 +++--- libgrive/src/drive/State.cc | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 4010cab2..f2852115 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -252,7 +252,7 @@ void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_sync ) void Resource::FromLocal( const DateTime& last_sync ) { fs::path path = Path() ; - assert( fs::exists( path ) ) ; + //assert( fs::exists( path ) ) ; // root folder is always in sync if ( !IsRoot() ) @@ -270,8 +270,8 @@ void Resource::FromLocal( const DateTime& last_sync ) m_state = ( m_mtime > last_sync ? local_new : remote_deleted ) ; m_name = path.filename().string() ; - m_kind = fs::is_directory(path) ? "folder" : "file" ; - m_md5 = fs::is_directory(path) ? "" : crypt::MD5::Get( path ) ; + //m_kind = fs::is_directory(path) ? "folder" : "file" ; + m_md5 = IsFolder() ? "" : crypt::MD5::Get( path ) ; } assert( m_state != unknown ) ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 124adecc..6c2ba611 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -84,19 +84,20 @@ void State::FromLocal( const fs::path& p, Resource* folder ) else { + bool is_dir = fs::is_directory(i->path()); // if the Resource object of the child already exists, it should // have been so no need to do anything here Resource *c = folder->FindChild( fname ) ; if ( c == 0 ) { - c = new Resource( fname, fs::is_directory(i->path()) ? "folder" : "file" ) ; + c = new Resource( fname, is_dir ? "folder" : "file" ) ; folder->AddChild( c ) ; m_res.Insert( c ) ; } c->FromLocal( m_last_sync ) ; - if ( fs::is_directory( i->path() ) ) + if ( is_dir ) FromLocal( *i, c ) ; } } From 986ab4acc6f0274d8d2cf54e52e80e0ce775ddee Mon Sep 17 00:00:00 2001 From: Visa Putkinen Date: Sun, 24 Nov 2013 14:34:52 +0200 Subject: [PATCH 059/166] Log XML on parse failure --- libgrive/src/xml/TreeBuilder.cc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/libgrive/src/xml/TreeBuilder.cc b/libgrive/src/xml/TreeBuilder.cc index c1ad385a..9d57501b 100644 --- a/libgrive/src/xml/TreeBuilder.cc +++ b/libgrive/src/xml/TreeBuilder.cc @@ -21,6 +21,7 @@ #include "Error.hh" #include "Node.hh" +#include "util/log/Log.hh" #include @@ -72,8 +73,10 @@ void TreeBuilder::ParseData( const char *data, std::size_t count, bool last ) { is_new = false ; - if ( ::XML_Parse( m_impl->psr, data, count, last ) == 0 ) + if ( ::XML_Parse( m_impl->psr, data, count, last ) == 0 ) { + Log("Error parsing XML: %1%", data, log::error); BOOST_THROW_EXCEPTION( Error() << ExpatApiError("XML_Parse") ); + } } Node TreeBuilder::Parse( const std::string& xml ) From 645bb2e7d408718b0b457add629d6d34843cbaef Mon Sep 17 00:00:00 2001 From: Visa Putkinen Date: Tue, 26 Nov 2013 23:00:42 +0200 Subject: [PATCH 060/166] Add delay before auth token refresh This is sometimes necessary to prevent too frequent requests. --- libgrive/src/protocol/AuthAgent.cc | 1 + 1 file changed, 1 insertion(+) diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 745f274c..6a329ceb 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -152,6 +152,7 @@ bool AuthAgent::CheckRetry( long response ) Log( "resquest failed due to auth token expired: %1%. refreshing token", response, log::warning ) ; + os::Sleep( 5 ) ; m_auth.Refresh() ; return true ; } From 84785ec4731b7a87b844a2339c400ab2e2e9d0ae Mon Sep 17 00:00:00 2001 From: Visa Putkinen Date: Mon, 25 Nov 2013 00:07:27 +0200 Subject: [PATCH 061/166] Fix hang when upload receives HTTP 500 When an uploading PUT request got a HTTP 500 as reponse, grive hanged forever inside libcurl. This was because the File parameter was not rewound to 0 position on retry. The XmlResponse had to be cleared as well. Rewinding the File and clearing the XmlResponse were not enough to fix the problem, because when retrying after 500, HTTP 410 Gone or 412 Precondition failed is often received, and CheckHttpResponse would throw an exception that crashes grive. Therefore, I implemented a retry logic to Resource::Upload that retries the whole upload transaction if 410 or 412 was received. --- libgrive/src/drive/Resource.cc | 54 +++++++++++++++++++++--------- libgrive/src/http/XmlResponse.cc | 5 +++ libgrive/src/protocol/AuthAgent.cc | 20 +++++++++-- 3 files changed, 61 insertions(+), 18 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index f2852115..7a7a2562 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -36,6 +36,7 @@ #include "xml/Node.hh" #include "xml/NodeSet.hh" #include "xml/String.hh" +#include "xml/TreeBuilder.hh" #include #include @@ -599,23 +600,44 @@ bool Resource::Upload( % xml::Escape(m_name) ).str() ; - http::StringResponse str ; - if ( post ) - http->Post( link, meta, &str, hdr ) ; - else - http->Put( link, meta, &str, hdr ) ; - - http::Header uphdr ; - uphdr.Add( "Expect:" ) ; - uphdr.Add( "Accept:" ) ; + bool retrying=false; + while ( true ) { + if ( retrying ) { + file.Seek( 0, SEEK_SET ); + os::Sleep( 5 ); + } - // the content upload URL is in the "Location" HTTP header - std::string uplink = http->RedirLocation() ; - http::XmlResponse xml ; - - http->Put( uplink, &file, &xml, uphdr ) ; - AssignIDs( Entry( xml.Response() ) ) ; - m_mtime = Entry(xml.Response()).MTime(); + http::StringResponse str ; + if ( post ) + http->Post( link, meta, &str, hdr ) ; + else + http->Put( link, meta, &str, hdr ) ; + + http::Header uphdr ; + uphdr.Add( "Expect:" ) ; + uphdr.Add( "Accept:" ) ; + + // the content upload URL is in the "Location" HTTP header + std::string uplink = http->RedirLocation() ; + http::XmlResponse xml ; + + long http_code = 0; + http_code = http->Put( uplink, &file, &xml, uphdr ) ; + + if ( http_code == 410 || http_code == 412 ) { + Log( "request failed with %1%, retrying whole upload in 5s", http_code, + log::warning ) ; + retrying = true; + continue; + } + + if ( retrying ) + Log( "upload succeeded on retry", log::warning ); + Entry responseEntry = Entry( xml.Response() ); + AssignIDs( responseEntry ) ; + m_mtime = responseEntry.MTime(); + break; + } return true ; } diff --git a/libgrive/src/http/XmlResponse.cc b/libgrive/src/http/XmlResponse.cc index b25f1c4d..3df42f94 100644 --- a/libgrive/src/http/XmlResponse.cc +++ b/libgrive/src/http/XmlResponse.cc @@ -28,6 +28,11 @@ XmlResponse::XmlResponse() : m_tb( new xml::TreeBuilder ) { } +void XmlResponse::Clear() +{ + m_tb.reset(new xml::TreeBuilder); +} + std::size_t XmlResponse::Write( const char *data, std::size_t count ) { m_tb->ParseData( data, count ) ; diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 6a329ceb..30a97223 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -21,8 +21,10 @@ #include "http/Error.hh" #include "http/Header.hh" +#include "http/XmlResponse.hh" #include "util/log/Log.hh" #include "util/OS.hh" +#include "util/File.hh" #include @@ -69,8 +71,22 @@ long AuthAgent::Put( Header auth = AppendHeader(hdr) ; long response ; - while ( CheckRetry( - response = m_agent->Put( url, file, dest, AppendHeader(hdr) ) ) ) ; + bool keepTrying = true; + while ( keepTrying ) { + response = m_agent->Put( url, file, dest, auth ); + keepTrying = CheckRetry( response ); + if ( keepTrying ) { + file->Seek( 0, SEEK_SET ); + XmlResponse *xmlResponse = dynamic_cast(dest); + if( xmlResponse ) + xmlResponse->Clear(); + } + } + + // On 410 Gone or 412 Precondition failed, recovery may be possible so don't + // throw an exception + if ( response == 410 || response == 412 ) + return response; return CheckHttpResponse(response, url, auth) ; } From 3775572f4650797e7eeab0ba23e27f1f15f01cd5 Mon Sep 17 00:00:00 2001 From: Visa Putkinen Date: Mon, 25 Nov 2013 00:14:29 +0200 Subject: [PATCH 062/166] Retry upload on XML error instead of crashing Sometimes the Google Drive API sends malformed XML which crashes grive. This patch adds a simple try-catch to Resource::Upload that retries the upload if an XML exception is thrown from AuthAgent::Put. --- libgrive/src/drive/Resource.cc | 36 ++++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 7a7a2562..2da7859e 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -607,11 +607,23 @@ bool Resource::Upload( os::Sleep( 5 ); } - http::StringResponse str ; - if ( post ) - http->Post( link, meta, &str, hdr ) ; - else - http->Put( link, meta, &str, hdr ) ; + try { + http::StringResponse str ; + if ( post ) + http->Post( link, meta, &str, hdr ) ; + else + http->Put( link, meta, &str, hdr ) ; + } catch ( Error &e ) { + std::string const *info = boost::get_error_info(e); + if ( info && (*info == "XML_Parse") ) { + Log( "Error parsing pre-upload response XML, retrying whole upload in 5s", + log::warning ); + retrying = true; + continue; + } else { + throw e; + } + } http::Header uphdr ; uphdr.Add( "Expect:" ) ; @@ -622,7 +634,19 @@ bool Resource::Upload( http::XmlResponse xml ; long http_code = 0; - http_code = http->Put( uplink, &file, &xml, uphdr ) ; + try { + http_code = http->Put( uplink, &file, &xml, uphdr ) ; + } catch ( Error &e ) { + std::string const *info = boost::get_error_info(e); + if ( info && (*info == "XML_Parse") ) { + Log( "Error parsing response XML, retrying whole upload in 5s", + log::warning ); + retrying = true; + continue; + } else { + throw e; + } + } if ( http_code == 410 || http_code == 412 ) { Log( "request failed with %1%, retrying whole upload in 5s", http_code, From 459313ecf502ae8b95e791dea552b46774c0a48d Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 23 Jul 2014 14:41:40 +0400 Subject: [PATCH 063/166] Find YAJL via pkg-config --- libgrive/CMakeLists.txt | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 3cc15742..28c3cbd3 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -12,6 +12,9 @@ find_package(CppUnit) find_package(Iberty) find_package(ZLIB) +find_package(PkgConfig) +pkg_check_modules(YAJL REQUIRED yajl) + # additional headers if build unit tests IF ( CPPUNIT_FOUND ) set( OPT_INCS ${CPPUNIT_INCLUDE_DIR} ) @@ -42,6 +45,7 @@ include_directories( ${libgrive_SOURCE_DIR}/test ${GDBM_INCLUDE_DIR} ${OPT_INCS} + ${YAJL_INCLUDE_DIRS} ) file(GLOB DRIVE_HEADERS @@ -80,7 +84,7 @@ add_definitions( add_library( grive STATIC ${LIBGRIVE_SRC} ${OPT_SRC} ) target_link_libraries( grive - yajl + ${YAJL_LIBRARIES} ${CURL_LIBRARIES} ${JSONC_LIBRARY} ${LIBGCRYPT_LIBRARIES} From 400d9e0c0ff150347aa043209d11afce5337b72e Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 23 Jul 2014 14:47:16 +0400 Subject: [PATCH 064/166] Fix long/int cast errors --- libgrive/src/drive/State.cc | 4 ++-- libgrive/src/drive/State.hh | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index cf0c30bd..de53a88e 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -257,8 +257,8 @@ void State::Read( const fs::path& filename ) void State::Write( const fs::path& filename ) const { Json last_sync ; - last_sync.Add( "sec", Json(m_last_sync.Sec() ) ); - last_sync.Add( "nsec", Json(m_last_sync.NanoSec() ) ); + last_sync.Add( "sec", Json( (int)m_last_sync.Sec() ) ); + last_sync.Add( "nsec", Json( (unsigned)m_last_sync.NanoSec() ) ); Json result ; result.Add( "last_sync", last_sync ) ; diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index 98cce39a..af79f8da 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -78,7 +78,7 @@ private : private : ResourceTree m_res ; DateTime m_last_sync ; - long m_cstamp ; + int m_cstamp ; std::string m_dir ; std::vector m_unresolved ; From 94efea11dd1735786d63ec1de3aaa385b2fd3c1d Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 23 Jul 2014 14:48:22 +0400 Subject: [PATCH 065/166] Do not try to install doc/grive.1 from bgrive cmake --- bgrive/CMakeLists.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/bgrive/CMakeLists.txt b/bgrive/CMakeLists.txt index 074ab383..7bb32692 100644 --- a/bgrive/CMakeLists.txt +++ b/bgrive/CMakeLists.txt @@ -40,4 +40,3 @@ set_target_properties( bgrive_executable ) install(TARGETS bgrive_executable RUNTIME DESTINATION bin) -install(FILES doc/grive.1 DESTINATION share/man/man1 ) From c0c89c306cf6f9d0c12604183d625426e7b8b1b6 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 14 May 2015 01:01:52 +0300 Subject: [PATCH 066/166] Remove useless swap() code --- libgrive/src/drive/Entry.cc | 23 ----------------------- libgrive/src/drive/Entry.hh | 2 -- libgrive/src/drive/Resource.cc | 28 ---------------------------- libgrive/src/drive/Resource.hh | 8 -------- libgrive/src/drive/ResourceTree.cc | 12 ------------ libgrive/src/drive/ResourceTree.hh | 3 --- 6 files changed, 76 deletions(-) diff --git a/libgrive/src/drive/Entry.cc b/libgrive/src/drive/Entry.cc index 23165d89..506295e1 100644 --- a/libgrive/src/drive/Entry.cc +++ b/libgrive/src/drive/Entry.cc @@ -152,29 +152,6 @@ std::string Entry::CreateLink() const return m_create_link ; } -void Entry::Swap( Entry& e ) -{ - m_title.swap( e.m_title ) ; - m_filename.swap( e.m_filename ) ; - m_kind.swap( e.m_kind ) ; - m_md5.swap( e.m_md5 ) ; - m_etag.swap( e.m_etag ) ; - m_resource_id.swap( e.m_resource_id ) ; - - m_parent_hrefs.swap( e.m_parent_hrefs ) ; - - m_self_href.swap( e.m_self_href ) ; - m_alt_self.swap( e.m_alt_self ) ; - m_content_src.swap( e.m_content_src ) ; - m_edit_link.swap( e.m_edit_link ) ; - m_create_link.swap( e.m_create_link ) ; - - m_mtime.Swap( e.m_mtime ) ; - - std::swap( m_change_stamp, e.m_change_stamp ) ; - std::swap( m_is_removed, e.m_is_removed ) ; -} - long Entry::ChangeStamp() const { return m_change_stamp ; diff --git a/libgrive/src/drive/Entry.hh b/libgrive/src/drive/Entry.hh index b7e9293a..a059b4f0 100644 --- a/libgrive/src/drive/Entry.hh +++ b/libgrive/src/drive/Entry.hh @@ -71,8 +71,6 @@ public : const std::vector& ParentHrefs() const ; - void Swap( Entry& e ) ; - void Update( const xml::Node& entry ) ; private : diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 4010cab2..89745086 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -314,26 +314,6 @@ void Resource::AddChild( Resource *child ) m_child.push_back( child ) ; } -void Resource::Swap( Resource& coll ) -{ - m_name.swap( coll.m_name ) ; - m_kind.swap( coll.m_kind ) ; - m_md5.swap( coll.m_md5 ) ; - m_etag.swap( coll.m_etag ) ; - m_id.swap( coll.m_id ) ; - - m_href.swap( coll.m_href ) ; - m_content.swap( coll.m_content ) ; - m_edit.swap( coll.m_edit ) ; - m_create.swap( coll.m_create ) ; - - m_mtime.Swap( coll.m_mtime ) ; - - std::swap( m_parent, coll.m_parent ) ; - m_child.swap( coll.m_child ) ; - std::swap( m_state, coll.m_state ) ; -} - bool Resource::IsFolder() const { return m_kind == "folder" ; @@ -669,11 +649,3 @@ bool Resource::HasID() const } } } // end of namespace - -namespace std -{ - void swap( gr::v1::Resource& c1, gr::v1::Resource& c2 ) - { - c1.Swap( c2 ) ; - } -} diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 46096807..190518e3 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -57,9 +57,6 @@ public : Resource(const fs::path& root_folder) ; Resource( const std::string& name, const std::string& kind ) ; - // default copy ctor & op= are fine - void Swap( Resource& coll ) ; - bool IsFolder() const ; std::string Name() const ; @@ -162,8 +159,3 @@ private : } ; } } // end of namespace gr::v1 - -namespace std -{ - void swap( gr::v1::Resource& c1, gr::v1::Resource& c2 ) ; -} diff --git a/libgrive/src/drive/ResourceTree.cc b/libgrive/src/drive/ResourceTree.cc index 4a8a6daf..d806232b 100644 --- a/libgrive/src/drive/ResourceTree.cc +++ b/libgrive/src/drive/ResourceTree.cc @@ -80,18 +80,6 @@ const Resource* ResourceTree::Root() const return m_root ; } -void ResourceTree::Swap( ResourceTree& fs ) -{ - m_set.swap( fs.m_set ) ; -} - -ResourceTree& ResourceTree::operator=( const ResourceTree& fs ) -{ - ResourceTree tmp( fs ) ; - Swap( tmp ) ; - return *this ; -} - Resource* ResourceTree::FindByHref( const std::string& href ) { // for the resource that not yet have href (e.g. not yet fetched from server) diff --git a/libgrive/src/drive/ResourceTree.hh b/libgrive/src/drive/ResourceTree.hh index 212141a8..6bd1501d 100644 --- a/libgrive/src/drive/ResourceTree.hh +++ b/libgrive/src/drive/ResourceTree.hh @@ -70,9 +70,6 @@ public : ResourceTree( const ResourceTree& fs ) ; ~ResourceTree( ) ; - void Swap( ResourceTree& fs ) ; - ResourceTree& operator=( const ResourceTree& fs ) ; - Resource* FindByHref( const std::string& href ) ; const Resource* FindByHref( const std::string& href ) const ; From f16f2122818be1d23e4e7bdf306f06d5e377d500 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 14 May 2015 01:20:35 +0300 Subject: [PATCH 067/166] use pkg-config also for finding json-c (fixes paths on debian) --- cmake/Modules/FindJSONC.cmake | 30 ------------------------------ libgrive/CMakeLists.txt | 5 +++-- libgrive/src/protocol/Json.cc | 4 ++-- 3 files changed, 5 insertions(+), 34 deletions(-) delete mode 100644 cmake/Modules/FindJSONC.cmake diff --git a/cmake/Modules/FindJSONC.cmake b/cmake/Modules/FindJSONC.cmake deleted file mode 100644 index f72e8ea1..00000000 --- a/cmake/Modules/FindJSONC.cmake +++ /dev/null @@ -1,30 +0,0 @@ -# - Find JSON-C -# This module finds an installed JSON-C package. -# -# It sets the following variables: -# JSONC_FOUND - Set to false, or undefined, if JSON-C isn't found. -# JSONC_INCLUDE_DIR - The JSON-C include directory. -# JSONC_LIBRARY - The JSON-C library to link against. - -FIND_PATH(JSONC_INCLUDE_DIR json/json.h) -FIND_LIBRARY(JSONC_LIBRARY NAMES json) - -IF (JSONC_INCLUDE_DIR AND JSONC_LIBRARY) - SET(JSONC_FOUND TRUE) -ENDIF (JSONC_INCLUDE_DIR AND JSONC_LIBRARY) - -IF (JSONC_FOUND) - - # show which JSON-C was found only if not quiet - IF (NOT JSONC_FIND_QUIETLY) - MESSAGE(STATUS "Found JSON-C: ${JSONC_LIBRARY}") - ENDIF (NOT JSONC_FIND_QUIETLY) - -ELSE (JSONC_FOUND) - - # fatal error if JSON-C is required but not found - IF (JSONC_FIND_REQUIRED) - MESSAGE(FATAL_ERROR "Could not find JSON-C") - ENDIF (JSONC_FIND_REQUIRED) - -ENDIF (JSONC_FOUND) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 28c3cbd3..a5a3dd7a 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -3,7 +3,6 @@ project(libgrive) set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/") find_package(LibGcrypt REQUIRED) -find_package(JSONC REQUIRED) find_package(CURL REQUIRED) find_package(EXPAT REQUIRED) find_package(Boost 1.40.0 COMPONENTS program_options filesystem unit_test_framework system REQUIRED) @@ -13,6 +12,7 @@ find_package(Iberty) find_package(ZLIB) find_package(PkgConfig) +pkg_check_modules(JSONC REQUIRED json-c) pkg_check_modules(YAJL REQUIRED yajl) # additional headers if build unit tests @@ -45,6 +45,7 @@ include_directories( ${libgrive_SOURCE_DIR}/test ${GDBM_INCLUDE_DIR} ${OPT_INCS} + ${JSONC_INCLUDE_DIRS} ${YAJL_INCLUDE_DIRS} ) @@ -86,7 +87,7 @@ add_library( grive STATIC ${LIBGRIVE_SRC} ${OPT_SRC} ) target_link_libraries( grive ${YAJL_LIBRARIES} ${CURL_LIBRARIES} - ${JSONC_LIBRARY} + ${JSONC_LIBRARIES} ${LIBGCRYPT_LIBRARIES} ${GDBM_LIBRARIES} ${Boost_LIBRARIES} diff --git a/libgrive/src/protocol/Json.cc b/libgrive/src/protocol/Json.cc index 4c115d6a..5abeaf26 100644 --- a/libgrive/src/protocol/Json.cc +++ b/libgrive/src/protocol/Json.cc @@ -29,8 +29,8 @@ #pragma warning(push) #pragma warning(disable: 4005) #endif -#include -#include +#include +#include #ifdef _MSC_VER #pragma warning(pop) #endif From 12774a952a2ef7534b0750966827c09cb404f5e2 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 14 May 2015 01:22:37 +0300 Subject: [PATCH 068/166] Remove bgrive (dead and unused code) --- CMakeLists.txt | 1 - bgrive/CMakeLists.txt | 42 ------------- bgrive/src/DriveModel.cc | 116 ---------------------------------- bgrive/src/DriveModel.hh | 61 ------------------ bgrive/src/MainWnd.cc | 58 ----------------- bgrive/src/MainWnd.hh | 56 ----------------- bgrive/src/main.cc | 62 ------------------ bgrive/ui/MainWindow.ui | 132 --------------------------------------- 8 files changed, 528 deletions(-) delete mode 100644 bgrive/CMakeLists.txt delete mode 100644 bgrive/src/DriveModel.cc delete mode 100644 bgrive/src/DriveModel.hh delete mode 100644 bgrive/src/MainWnd.cc delete mode 100644 bgrive/src/MainWnd.hh delete mode 100644 bgrive/src/main.cc delete mode 100644 bgrive/ui/MainWindow.ui diff --git a/CMakeLists.txt b/CMakeLists.txt index 48f1460b..def7ca00 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -9,4 +9,3 @@ add_definitions( -D_FILE_OFFSET_BITS=64 ) add_subdirectory( libgrive ) add_subdirectory( grive ) -add_subdirectory( bgrive ) diff --git a/bgrive/CMakeLists.txt b/bgrive/CMakeLists.txt deleted file mode 100644 index 7bb32692..00000000 --- a/bgrive/CMakeLists.txt +++ /dev/null @@ -1,42 +0,0 @@ -project( bgrive ) - -find_package(Qt4 REQUIRED) -find_package(Boost REQUIRED) -INCLUDE(${QT_USE_FILE}) - -include_directories( - ${bgrive_SOURCE_DIR}/../libgrive/src - ${CMAKE_CURRENT_SOURCE_DIR} - ${CMAKE_CURRENT_BINARY_DIR} -) - -file (GLOB BGRIVE_EXE_SRC - ${bgrive_SOURCE_DIR}/src/*.cc -) - -file (GLOB BGRIVE_UI - ${bgrive_SOURCE_DIR}/ui/*.ui -) - -QT4_WRAP_UI(BGRIVE_UI_SRCS ${BGRIVE_UI}) -QT4_WRAP_CPP(BGRIVE_MOC_SRCS - src/MainWnd.hh ) - -add_executable( bgrive_executable - ${BGRIVE_EXE_SRC} - ${BGRIVE_UI_SRCS} - ${BGRIVE_MOC_SRCS} -) - -target_link_libraries( bgrive_executable - ${Boost_LIBRARIES} - ${QT_QTMAIN_LIBRARY} - ${QT_LIBRARIES} - grive -) - -set_target_properties( bgrive_executable - PROPERTIES OUTPUT_NAME bgrive -) - -install(TARGETS bgrive_executable RUNTIME DESTINATION bin) diff --git a/bgrive/src/DriveModel.cc b/bgrive/src/DriveModel.cc deleted file mode 100644 index bc3bcf43..00000000 --- a/bgrive/src/DriveModel.cc +++ /dev/null @@ -1,116 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#include "DriveModel.hh" - -#include "drive2/Resource.hh" - -#include - -namespace gr { - -using namespace v2; - -DriveModel::DriveModel( http::Agent *agent ) -{ - m_drv.Refresh( agent ) ; -} - -Qt::ItemFlags DriveModel::flags( const QModelIndex& ) const -{ - return Qt::ItemIsEnabled | Qt::ItemIsSelectable ; -} - -QVariant DriveModel::data( const QModelIndex& index, int role ) const -{ - const Resource *res = Res(index) ; - if ( role == Qt::DisplayRole && res != 0 ) - { - switch ( index.column() ) - { - case 0: return QString::fromUtf8(res->Title().c_str()) ; - default: break ; - } - } - - return QVariant() ; -} - -QVariant DriveModel::headerData( int section, Qt::Orientation orientation, int role ) const -{ - return role == Qt::DisplayRole ? QString("header") : QVariant() ; -} - -int DriveModel::rowCount( const QModelIndex& parent ) const -{ - return Res(parent)->ChildCount() ; -} - -int DriveModel::columnCount( const QModelIndex& parent ) const -{ - return 1 ; -} - -bool DriveModel::hasChildren( const QModelIndex& parent ) const -{ - return Res(parent)->ChildCount() > 0 ; -} - -QModelIndex DriveModel::index( int row, int column, const QModelIndex& parent_idx ) const -{ - const Resource *parent = Res(parent_idx) ; - - // check out-of-bound - if ( parent != 0 && static_cast(row) >= parent->ChildCount() ) - BOOST_THROW_EXCEPTION( - Exception() - << InvalidRow_( row ) - << ResourceName_( parent->Title() ) - ) ; - - - return createIndex( row, column, const_cast(m_drv.Child(parent, row)) ) ; -} - -const Resource* DriveModel::Res( const QModelIndex& idx ) const -{ - return idx.isValid() - ? reinterpret_cast(idx.internalPointer()) - : m_drv.Root() ; -} - -QModelIndex DriveModel::parent( const QModelIndex& idx ) const -{ - // if I am root, my parent is myself - const Resource *res = Res(idx) ; - if ( res == m_drv.Root() ) - return QModelIndex() ; - - // if my parent is root, return model index of root (i.e. QModelIndex()) - const Resource *parent = m_drv.Parent(res) ; - if ( parent == 0 || parent == m_drv.Root() || idx.column() != 0 ) - return QModelIndex() ; - - // check grand-parent to know the row() of my parent - const Resource *grand = m_drv.Parent(parent) ; - return createIndex( grand->Index(parent->ID()), 0, const_cast(parent) ) ; -} - -} // end of namespace diff --git a/bgrive/src/DriveModel.hh b/bgrive/src/DriveModel.hh deleted file mode 100644 index f39847f4..00000000 --- a/bgrive/src/DriveModel.hh +++ /dev/null @@ -1,61 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#pragma once - -#include - -#include "drive2/Drive.hh" -#include "util/Exception.hh" - -namespace gr { - -namespace http -{ - class Agent ; -} - -class DriveModel : public QAbstractItemModel -{ -public : - typedef boost::error_info InvalidRow_ ; - typedef boost::error_info ResourceName_ ; - -public : - DriveModel( http::Agent *agent ) ; - - // QAbstractItemModel overrides - Qt::ItemFlags flags( const QModelIndex & index ) const ; - QVariant data( const QModelIndex& index, int role ) const ; - QVariant headerData( int section, Qt::Orientation orientation, int role ) const ; - int rowCount( const QModelIndex& parent ) const ; - int columnCount( const QModelIndex& parent ) const ; - bool hasChildren ( const QModelIndex& parent ) const ; - QModelIndex index( int row, int column, const QModelIndex& parent ) const ; - QModelIndex parent( const QModelIndex& idx ) const ; - - const v2::Resource* Res( const QModelIndex& idx ) const ; - -private : - v2::Drive m_drv ; -} ; - -} // end of namespace - diff --git a/bgrive/src/MainWnd.cc b/bgrive/src/MainWnd.cc deleted file mode 100644 index 28d4e088..00000000 --- a/bgrive/src/MainWnd.cc +++ /dev/null @@ -1,58 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#include "MainWnd.hh" - -#include "drive2/Resource.hh" - -#include - -#include - -namespace gr { - -using namespace v2 ; - -MainWnd::MainWnd( http::Agent *agent ) : - m_drive( agent ) -{ - m_ui.setupUi(this) ; - m_ui.m_dir->setModel( &m_drive ) ; - - connect( - m_ui.m_dir, SIGNAL(activated(const QModelIndex&)), - this, SLOT(OnClick(const QModelIndex&)) - ) ; -} - -void MainWnd::OnClick( const QModelIndex& index ) -{ - const Resource *res = m_drive.Res(index) ; - if ( res != 0 ) - ShowResource( res ) ; -} - -void MainWnd::ShowResource( const v2::Resource *res ) -{ - m_ui.m_title->setText( QString::fromUtf8(res->Title().c_str()) ) ; - m_ui.m_mime_type->setText( QString::fromUtf8(res->Mime().c_str()) ) ; -} - -} // end of namespace diff --git a/bgrive/src/MainWnd.hh b/bgrive/src/MainWnd.hh deleted file mode 100644 index fe9401b6..00000000 --- a/bgrive/src/MainWnd.hh +++ /dev/null @@ -1,56 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#pragma once - -#include -#include "ui_MainWindow.h" - -#include "DriveModel.hh" - -class QModelIndex ; - -namespace gr { - -namespace http -{ - class Agent ; -} - -class MainWnd : public QMainWindow -{ - Q_OBJECT - -public : - MainWnd( http::Agent *agent ) ; - -private : - void ShowResource( const v2::Resource *res ) ; - -public slots : - void OnClick( const QModelIndex& index ) ; - -private : - Ui::MainWindow m_ui ; - DriveModel m_drive ; -} ; - -} // end of namespace - diff --git a/bgrive/src/main.cc b/bgrive/src/main.cc deleted file mode 100644 index ea578ec9..00000000 --- a/bgrive/src/main.cc +++ /dev/null @@ -1,62 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#include "MainWnd.hh" - -#include -#include - -#include "drive2/Drive.hh" - -#include "http/CurlAgent.hh" -#include "http/Header.hh" -#include "protocol/JsonResponse.hh" - -#include "protocol/Json.hh" -#include "protocol/OAuth2.hh" -#include "protocol/AuthAgent.hh" - -#include "util/File.hh" - -#include -#include - -const std::string client_id = "22314510474.apps.googleusercontent.com" ; -const std::string client_secret = "bl4ufi89h-9MkFlypcI7R785" ; - -using namespace gr ; -using namespace gr::v2 ; - -int main( int argc, char **argv ) -{ - File file( ".grive" ) ; - Json cfg = Json::Parse( &file ) ; - - std::string refresh_token = cfg["refresh_token"].Str() ; - OAuth2 token( refresh_token, client_id, client_secret ) ; - - AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; - - QApplication app( argc, argv ) ; - MainWnd wnd( &agent ) ; - wnd.show(); - - return app.exec() ; -} diff --git a/bgrive/ui/MainWindow.ui b/bgrive/ui/MainWindow.ui deleted file mode 100644 index 61c99206..00000000 --- a/bgrive/ui/MainWindow.ui +++ /dev/null @@ -1,132 +0,0 @@ - - - MainWindow - - - - 0 - 0 - 800 - 600 - - - - Grive - - - - - - - Qt::Horizontal - - - - - 0 - 0 - - - - - - - - - Title: - - - - - - - - - - - - - - - - - Filename: - - - - - - - - - - Mime type: - - - - - - - - - - Last Updated: - - - - - - - - - - - - - - - - - - - 0 - 0 - 800 - 23 - - - - - &File - - - - - - - - - E&xit - - - - - - - m_action_exit - activated() - MainWindow - close() - - - -1 - -1 - - - 399 - 299 - - - - - From 5bc503279a139f0cfe6d7f1a4e052a3cdd1909b7 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 16 May 2015 00:27:59 +0300 Subject: [PATCH 069/166] Remove JSON-C json wrappers, use YAJL for everything --- README | 2 +- grive/src/main.cc | 4 +- libgrive/CMakeLists.txt | 3 - libgrive/src/drive/Drive.cc | 2 +- libgrive/src/drive/Drive.hh | 6 +- libgrive/src/drive/Resource.cc | 6 +- libgrive/src/drive/Resource.hh | 6 +- libgrive/src/drive/ResourceTree.cc | 1 - libgrive/src/drive/ResourceTree.hh | 2 - libgrive/src/drive/State.cc | 26 +- libgrive/src/drive/State.hh | 6 +- libgrive/src/json/JsonParser.cc | 26 +- libgrive/src/json/JsonParser.hh | 8 +- libgrive/src/protocol/Json.cc | 527 -------------------------- libgrive/src/protocol/Json.hh | 144 ------- libgrive/src/protocol/JsonResponse.cc | 45 --- libgrive/src/protocol/JsonResponse.hh | 46 --- libgrive/src/protocol/OAuth2.cc | 9 +- libgrive/src/util/Config.cc | 35 +- libgrive/src/util/Config.hh | 14 +- libgrive/test/btest/JsonValTest.cc | 4 +- libgrive/test/drive/StateTest.cc | 2 +- libgrive/test/util/ConfigTest.cc | 2 +- package/fedora16/grive.spec | 2 +- 24 files changed, 94 insertions(+), 834 deletions(-) delete mode 100644 libgrive/src/protocol/Json.cc delete mode 100644 libgrive/src/protocol/Json.hh delete mode 100644 libgrive/src/protocol/JsonResponse.cc delete mode 100644 libgrive/src/protocol/JsonResponse.hh diff --git a/README b/README index a713a75c..7db8650d 100644 --- a/README +++ b/README @@ -19,7 +19,7 @@ Of course these will be added in the future, possibly the next release. You need the following libraries: -- json-c +- yajl - libcurl - libstdc++ - libgcrypt diff --git a/grive/src/main.cc b/grive/src/main.cc index b7b25852..b172f409 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -24,7 +24,7 @@ #include "http/CurlAgent.hh" #include "protocol/AuthAgent.hh" #include "protocol/OAuth2.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" #include "bfd/Backtrace.hh" #include "util/Exception.hh" @@ -164,7 +164,7 @@ int Main( int argc, char **argv ) token.Auth( code ) ; // save to config - config.Set( "refresh_token", Json( token.RefreshToken() ) ) ; + config.Set( "refresh_token", Val( token.RefreshToken() ) ) ; config.Save() ; } diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index a5a3dd7a..fd610532 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -12,7 +12,6 @@ find_package(Iberty) find_package(ZLIB) find_package(PkgConfig) -pkg_check_modules(JSONC REQUIRED json-c) pkg_check_modules(YAJL REQUIRED yajl) # additional headers if build unit tests @@ -45,7 +44,6 @@ include_directories( ${libgrive_SOURCE_DIR}/test ${GDBM_INCLUDE_DIR} ${OPT_INCS} - ${JSONC_INCLUDE_DIRS} ${YAJL_INCLUDE_DIRS} ) @@ -87,7 +85,6 @@ add_library( grive STATIC ${LIBGRIVE_SRC} ${OPT_SRC} ) target_link_libraries( grive ${YAJL_LIBRARIES} ${CURL_LIBRARIES} - ${JSONC_LIBRARIES} ${LIBGCRYPT_LIBRARIES} ${GDBM_LIBRARIES} ${Boost_LIBRARIES} diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index 39354484..c5e4eef3 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -51,7 +51,7 @@ namespace const std::string state_file = ".grive_state" ; } -Drive::Drive( http::Agent *agent, const Json& options ) : +Drive::Drive( http::Agent *agent, const Val& options ) : m_http ( agent ), m_root ( options["path"].Str() ), m_state ( m_root / state_file, options ), diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 0490d5c1..1f1e4758 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -22,7 +22,7 @@ #include "State.hh" #include "http/Header.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" #include "util/Exception.hh" #include @@ -42,7 +42,7 @@ class Entry ; class Drive { public : - Drive( http::Agent *agent, const Json& options ) ; + Drive( http::Agent *agent, const Val& options ) ; void DetectChanges() ; void Update() ; @@ -63,7 +63,7 @@ private : std::string m_resume_link ; fs::path m_root ; State m_state ; - Json m_options ; + Val m_options ; } ; } } // end of namespace diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 498a4268..a9d28b92 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -27,7 +27,7 @@ // #include "http/ResponseLog.hh" #include "http/StringResponse.hh" #include "http/XmlResponse.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" #include "util/CArray.hh" #include "util/Crypt.hh" #include "util/log/Log.hh" @@ -346,7 +346,7 @@ Resource* Resource::FindChild( const std::string& name ) } // try to change the state to "sync" -void Resource::Sync( http::Agent *http, DateTime& sync_time, const Json& options ) +void Resource::Sync( http::Agent *http, DateTime& sync_time, const Val& options ) { assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced @@ -363,7 +363,7 @@ void Resource::Sync( http::Agent *http, DateTime& sync_time, const Json& options boost::bind( &Resource::Sync, _1, http, boost::ref(sync_time), options ) ) ; } -void Resource::SyncSelf( http::Agent* http, const Json& options ) +void Resource::SyncSelf( http::Agent* http, const Val& options ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync assert( IsRoot() || http == 0 || fs::is_directory( m_parent->Path() ) ) ; diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 190518e3..57f9262c 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -34,7 +34,7 @@ namespace http class Agent ; } -class Json ; +class Val ; namespace v1 { @@ -77,7 +77,7 @@ public : void FromRemote( const Entry& remote, const DateTime& last_sync ) ; void FromLocal( const DateTime& last_sync ) ; - void Sync( http::Agent* http, DateTime& sync_time, const Json& options ) ; + void Sync( http::Agent* http, DateTime& sync_time, const Val& options ) ; // children access iterator begin() const ; @@ -136,7 +136,7 @@ private : void DeleteRemote( http::Agent* http ) ; void AssignIDs( const Entry& remote ) ; - void SyncSelf( http::Agent* http, const Json& options ) ; + void SyncSelf( http::Agent* http, const Val& options ) ; private : std::string m_name ; diff --git a/libgrive/src/drive/ResourceTree.cc b/libgrive/src/drive/ResourceTree.cc index d806232b..fa8af7e3 100644 --- a/libgrive/src/drive/ResourceTree.cc +++ b/libgrive/src/drive/ResourceTree.cc @@ -20,7 +20,6 @@ #include "ResourceTree.hh" #include "CommonUri.hh" -#include "protocol/Json.hh" #include "util/Destroy.hh" #include "util/log/Log.hh" diff --git a/libgrive/src/drive/ResourceTree.hh b/libgrive/src/drive/ResourceTree.hh index 6bd1501d..d0a391c9 100644 --- a/libgrive/src/drive/ResourceTree.hh +++ b/libgrive/src/drive/ResourceTree.hh @@ -30,8 +30,6 @@ namespace gr { -class Json ; - namespace v1 { namespace details diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 3b227c90..c1a17e56 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -27,13 +27,14 @@ #include "util/Crypt.hh" #include "util/File.hh" #include "util/log/Log.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" +#include "json/JsonParser.hh" #include namespace gr { namespace v1 { -State::State( const fs::path& filename, const Json& options ) : +State::State( const fs::path& filename, const Val& options ) : m_res ( options["path"].Str() ), m_dir ( options["dir"].Str() ), m_cstamp ( -1 ) @@ -41,7 +42,7 @@ State::State( const fs::path& filename, const Json& options ) : Read( filename ) ; // the "-f" option will make grive always thinks remote is newer - Json force ; + Val force ; if ( options.Get("force", force) && force.Bool() ) m_last_sync = DateTime() ; @@ -240,9 +241,10 @@ void State::Read( const fs::path& filename ) try { File file( filename ) ; - Json json = Json::Parse( &file ) ; - - Json last_sync = json["last_sync"] ; + + Val json = ParseJson( file ); + + Val last_sync = json["last_sync"] ; m_last_sync.Assign( last_sync["sec"].Int(), last_sync["nsec"].Int() ) ; @@ -257,19 +259,19 @@ void State::Read( const fs::path& filename ) void State::Write( const fs::path& filename ) const { - Json last_sync ; - last_sync.Add( "sec", Json( (int)m_last_sync.Sec() ) ); - last_sync.Add( "nsec", Json( (unsigned)m_last_sync.NanoSec() ) ); + Val last_sync ; + last_sync.Add( "sec", Val( (int)m_last_sync.Sec() ) ); + last_sync.Add( "nsec", Val( (unsigned)m_last_sync.NanoSec() ) ); - Json result ; + Val result ; result.Add( "last_sync", last_sync ) ; - result.Add( "change_stamp", Json(m_cstamp) ) ; + result.Add( "change_stamp", Val(m_cstamp) ) ; std::ofstream fs( filename.string().c_str() ) ; fs << result ; } -void State::Sync( http::Agent *http, const Json& options ) +void State::Sync( http::Agent *http, const Val& options ) { // set the last sync time from the time returned by the server for the last file synced // if the sync time hasn't changed (i.e. now files have been uploaded) diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index af79f8da..50492519 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -33,7 +33,7 @@ namespace http class Agent ; } -class Json ; +class Val ; namespace v1 { @@ -46,7 +46,7 @@ public : typedef ResourceTree::iterator iterator ; public : - explicit State( const fs::path& filename, const Json& options ) ; + explicit State( const fs::path& filename, const Val& options ) ; ~State() ; void FromLocal( const fs::path& p ) ; @@ -59,7 +59,7 @@ public : Resource* FindByHref( const std::string& href ) ; Resource* FindByID( const std::string& id ) ; - void Sync( http::Agent *http, const Json& options ) ; + void Sync( http::Agent *http, const Val& options ) ; iterator begin() ; iterator end() ; diff --git a/libgrive/src/json/JsonParser.cc b/libgrive/src/json/JsonParser.cc index 9ae63f48..50092cfa 100644 --- a/libgrive/src/json/JsonParser.cc +++ b/libgrive/src/json/JsonParser.cc @@ -114,11 +114,22 @@ namespace }; } -void JsonParser::Parse( const std::string& json, ValVisitor *callback ) +Val ParseJson( const std::string& json ) { - JsonParser parser( callback ) ; + ValBuilder b; + JsonParser parser( &b ) ; parser.Parse( json.c_str(), json.size() ) ; parser.Finish() ; + return b.Result(); +} + +Val ParseJson( DataStream &in ) +{ + ValBuilder b; + JsonParser parser( &b ) ; + parser.Parse( in ) ; + parser.Finish() ; + return b.Result(); } struct JsonParser::Impl @@ -159,6 +170,17 @@ void JsonParser::Parse( const char *str, std::size_t size ) } } +void JsonParser::Parse( DataStream &in ) +{ + char buf[1024] ; + std::size_t count = 0 ; + + while ( (count = in.Read( buf, sizeof(buf) ) ) > 0 ) + { + Parse( buf, count ); + } +} + void JsonParser::Finish() { if ( yajl_complete_parse(m_impl->hand) != yajl_status_ok ) diff --git a/libgrive/src/json/JsonParser.hh b/libgrive/src/json/JsonParser.hh index 27d6a4c8..ea9e09ed 100644 --- a/libgrive/src/json/JsonParser.hh +++ b/libgrive/src/json/JsonParser.hh @@ -20,7 +20,9 @@ #pragma once +#include "Val.hh" #include "util/Exception.hh" +#include "util/DataStream.hh" #include #include @@ -29,6 +31,9 @@ namespace gr { class ValVisitor ; +Val ParseJson( const std::string& json ) ; +Val ParseJson( DataStream &in ) ; + class JsonParser { public : @@ -36,12 +41,11 @@ public : typedef boost::error_info ParseErr_ ; typedef boost::error_info JsonText_ ; - static void Parse( const std::string& json, ValVisitor *callback ) ; - explicit JsonParser( ValVisitor *callback ) ; ~JsonParser() ; void Parse( const char *str, std::size_t size ) ; + void Parse( DataStream &in ) ; void Finish() ; private : diff --git a/libgrive/src/protocol/Json.cc b/libgrive/src/protocol/Json.cc deleted file mode 100644 index 5abeaf26..00000000 --- a/libgrive/src/protocol/Json.cc +++ /dev/null @@ -1,527 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#include "Json.hh" - -#include "util/DataStream.hh" - -// needs to include stdint.h before json-c to avoid macro re-def warning -#include - -// disable macro re-def warning for json-c headers -#ifdef _MSC_VER - #pragma warning(push) - #pragma warning(disable: 4005) -#endif -#include -#include -#ifdef _MSC_VER - #pragma warning(pop) -#endif - -#include -#include -#include -#include - -namespace gr { - -Json::Json( ) : - m_json( ::json_object_new_object() ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_object" ) - ) ; -} - -Json::Json( const char *str ) : - m_json( ::json_object_new_string( str ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_string" ) - << ValueErr( str ) - ) ; -} - -/** Note that json_object_new_string_len() is not used. -*/ -struct json_object* Json::InitStr( const char *str, std::size_t n ) -{ - struct json_object *j = ::json_object_new_string( str ) ; - if ( j == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_string_len" ) - << ValueErr( std::string(str, n) ) - ) ; - return j ; -} - -template <> -Json::Json( const std::string& str ) : - m_json( InitStr( str.c_str(), str.size() ) ) -{ -} - -template <> -Json::Json( const double& val ) : - m_json( ::json_object_new_double( val ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_double" ) - << ValueErr( val ) - ) ; -} - -template <> -Json::Json( const boost::int32_t& l ) : - m_json( ::json_object_new_int( l ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_int" ) - << ValueErr( l ) - ) ; -} - -template <> -Json::Json( const boost::int64_t& l ) : - m_json( ::json_object_new_int64( l ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_int64" ) - << ValueErr( l ) - ) ; -} - -template <> -Json::Json( const boost::uint32_t& l ) : - m_json( ::json_object_new_int( static_cast(l) ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_int" ) - << ValueErr( l ) - ) ; -} - -template <> -Json::Json( const boost::uint64_t& l ) : - m_json( ::json_object_new_int64( l ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_int64" ) - << ValueErr( l ) - ) ; -} - -template <> -Json::Json( const std::vector& arr ) : - m_json( ::json_object_new_array( ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << JsonCApi_( "json_object_new_array" ) ) ; - - for ( std::vector::const_iterator i = arr.begin() ; i != arr.end() ; ++i ) - Add( *i ) ; -} - -template <> -Json::Json( const bool& b ) : - m_json( ::json_object_new_boolean( b ) ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_new_boolean" ) - << ValueErr( b ) - ) ; -} - -template <> -Json::Json( const Object& obj ) : - m_json( ::json_object_new_object() ) -{ - if ( m_json == 0 ) - BOOST_THROW_EXCEPTION( Error() << JsonCApi_( "json_object_new_object" ) ) ; - - for ( Object::const_iterator i = obj.begin() ; i != obj.end() ; ++i ) - Add( i->first, i->second ) ; -} - -Json::Json( struct json_object *json, NotOwned ) : - m_json( json ) -{ - assert( m_json != 0 ) ; -} - -Json::Json( struct json_object *json ) : - m_json( json ) -{ - assert( json != 0 ) ; - ::json_object_get( m_json ) ; -} - -Json::Json( const Json& rhs ) : - m_json( rhs.m_json ) -{ - assert( m_json != 0 ) ; - ::json_object_get( m_json ) ; -} - -Json::~Json( ) -{ - assert( m_json != 0 ) ; - if ( m_json != 0 ) - ::json_object_put( m_json ) ; -} - -Json& Json::operator=( const Json& rhs ) -{ - Json tmp( rhs ) ; - Swap( tmp ) ; - return *this ; -} - -void Json::Swap( Json& other ) -{ - assert( m_json != 0 ) ; - assert( other.m_json != 0 ) ; - std::swap( m_json, other.m_json ) ; -} - -Json Json::operator[]( const std::string& key ) const -{ - assert( m_json != 0 ) ; - - struct json_object *j = 0 ; - if ( !::json_object_object_get_ex( m_json, key.c_str(), &j ) ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_object_get" ) - << KeyNotFound_( key ) - << Json_( ::json_object_to_json_string(m_json) ) ) ; - - assert( j != 0 ) ; - return Json( j ) ; -} - -Json Json::operator[]( const std::size_t& idx ) const -{ - assert( m_json != 0 ) ; - - struct json_object *j = ::json_object_array_get_idx( m_json, idx ) ; - if ( j == 0 ) - { - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_object_array_get_idx" ) - << OutOfRange_( idx ) - << Json_( ::json_object_to_json_string(m_json) ) ) ; - } - - return Json( j ) ; -} - -bool Json::Has( const std::string& key ) const -{ - assert( m_json != 0 ) ; - return ::json_object_object_get_ex( m_json, key.c_str(), 0 ) == TRUE ; -} - -bool Json::Get( const std::string& key, Json& json ) const -{ - assert( m_json != 0 ) ; - struct json_object *j = 0 ; - if ( ::json_object_object_get_ex( m_json, key.c_str(), &j ) ) - { - assert( j != 0 ) ; - - Json tmp( j ) ; - json.Swap( tmp ) ; - return true ; - } - else - return false ; -} - -void Json::Add( const std::string& key, const Json& json ) -{ - assert( m_json != 0 ) ; - assert( json.m_json != 0 ) ; - - ::json_object_get( json.m_json ) ; - ::json_object_object_add( m_json, key.c_str(), json.m_json ) ; -} - -void Json::Add( const Json& json ) -{ - assert( m_json != 0 ) ; - assert( json.m_json != 0 ) ; - - ::json_object_get( json.m_json ) ; - ::json_object_array_add( m_json, json.m_json ) ; -} - -bool Json::Bool() const -{ - assert( m_json != 0 ) ; - return ::json_object_get_boolean( m_json ) == TRUE ; -} - -template <> -bool Json::Is() const -{ - assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_boolean ) == TRUE ; -} - -template <> -bool Json::As() const -{ - return Bool() ; -} - -std::string Json::Str() const -{ - assert( m_json != 0 ) ; - return ::json_object_get_string( m_json ) ; -} - -template <> -bool Json::Is() const -{ - assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_string ) == TRUE ; -} - -template <> -std::string Json::As() const -{ - return Str() ; -} - -int Json::Int() const -{ - assert( m_json != 0 ) ; - return ::json_object_get_int( m_json ) ; -} - -template <> -bool Json::Is() const -{ - assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_int ) == TRUE ; -} - -template <> -boost::int32_t Json::As() const -{ - return Int() ; -} - -template <> -boost::uint32_t Json::As() const -{ - return static_cast(Int()) ; -} - -template <> -boost::int64_t Json::As() const -{ - return ::json_object_get_int64( m_json ) ; -} - -template <> -boost::uint64_t Json::As() const -{ - return ::json_object_get_int64( m_json ) ; -} - -std::ostream& operator<<( std::ostream& os, const Json& json ) -{ - assert( json.m_json != 0 ) ; - return os << ::json_object_to_json_string( json.m_json ) ; -} - -void Json::Write( DataStream *out ) const -{ - assert( out != 0 ) ; - - const char *str = ::json_object_to_json_string( m_json ) ; - out->Write( str, std::strlen(str) ) ; -} - -Json::Type Json::DataType() const -{ - assert( m_json != 0 ) ; - return static_cast( ::json_object_get_type( m_json ) ) ; -} - -Json::Object Json::AsObject() const -{ - Object result ; - - json_object_object_foreach( m_json, key, val ) - { - result.insert( Object::value_type( key, Json( val ) ) ) ; - } - - return result ; -} - -template <> -bool Json::Is() const -{ - assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_object ) == TRUE ; -} - -template <> -Json::Object Json::As() const -{ - return AsObject() ; -} - -Json::Array Json::AsArray() const -{ - std::size_t count = ::json_object_array_length( m_json ) ; - Array result ; - - for ( std::size_t i = 0 ; i < count ; ++i ) - result.push_back( Json( ::json_object_array_get_idx( m_json, i ) ) ) ; - - return result ; -} - -template <> -bool Json::Is() const -{ - assert( m_json != 0 ) ; - return ::json_object_is_type( m_json, json_type_array ) == TRUE ; -} - -template <> -Json::Array Json::As() const -{ - return AsArray() ; -} - -/// Finds an element in the array. -/// \pre "this" is an array -/// \return *this[i] if *this[i][key] == value -Json Json::FindInArray( const std::string& key, const std::string& value ) const -{ - std::size_t count = ::json_object_array_length( m_json ) ; - - for ( std::size_t i = 0 ; i < count ; ++i ) - { - Json item( ::json_object_array_get_idx( m_json, i ) ) ; - if ( item.Has(key) && item[key].Str() == value ) - return item ; - } - - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "Json::FindInArray" ) - << KeyNotFound_( key ) - << Value_(value) - ) ; - - // shut off compiler warnings - return Json() ; -} - -bool Json::FindInArray( const std::string& key, const std::string& value, Json& result ) const -{ - try - { - result = FindInArray( key, value ) ; - return true ; - } - catch ( Error& ) - { - } - return false ; -} - -Json Json::Parse( const std::string& str ) -{ - struct json_object *json = ::json_tokener_parse( str.c_str() ) ; - if ( json == 0 ) - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_tokener_parse" ) - << ValueErr( str ) - ) ; - - return Json( json, NotOwned() ) ; -} - -/// Parse a file. The file is loaded from file system. -/// \throw Error expt::ErrMsg contains a human-readable message describing the -/// error. -Json Json::Parse( DataStream *in ) -{ - assert( in != 0 ) ; - - struct json_tokener *tok = ::json_tokener_new() ; - struct json_object *json = 0 ; - - char buf[1024] ; - std::size_t count = 0 ; - - while ( (count = in->Read( buf, sizeof(buf) ) ) > 0 ) - { - json = ::json_tokener_parse_ex( tok, buf, count ) ; - - // check for parse error - if ( ::json_tokener_get_error(tok) == ::json_tokener_continue ) - break ; - } - - // save the error code and free the tokener before throwing exceptions - ::json_tokener_error err = ::json_tokener_get_error(tok) ; - ::json_tokener_free( tok ) ; tok = 0 ; - - if ( err != json_tokener_success || json == 0 ) - { - BOOST_THROW_EXCEPTION( - Error() - << JsonCApi_( "json_tokener_parse" ) - << ErrMsg_( ::json_tokener_error_desc(err) ) - ) ; - } - - return Json( json, NotOwned() ) ; -} - -} diff --git a/libgrive/src/protocol/Json.hh b/libgrive/src/protocol/Json.hh deleted file mode 100644 index 320f9d71..00000000 --- a/libgrive/src/protocol/Json.hh +++ /dev/null @@ -1,144 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include "util/Exception.hh" - -#include -#include -#include - -struct json_object ; - -namespace gr { - -class DataStream ; - -/*! \brief Simple wrapper around JSON-C objects. - - This class represents JSON-C objects, which can be integers, booleans, strings - double, arrays and object. -*/ -class Json -{ -public : - typedef std::map Object ; - typedef std::vector Array ; - - struct Error : virtual Exception {} ; - typedef boost::error_info Json_ ; - typedef boost::error_info OutOfRange_ ; - typedef boost::error_info KeyNotFound_ ; - typedef boost::error_info JsonCApi_ ; - typedef boost::error_info Value_ ; - typedef boost::error_info ErrMsg_ ; - - template - struct Val_ - { - typedef boost::error_info Err ; - } ; - -public : - template - explicit Json( const T& val ) ; - - template - explicit Json( const char (&str)[n] ) : - m_json( InitStr( str, n ) ) - { - } - - Json() ; - Json( const Json& rhs ) ; - Json( const char *str ) ; - ~Json() ; - - static Json Parse( const std::string& str ) ; - static Json Parse( DataStream *in ) ; - - Json operator[]( const std::string& key ) const ; - Json operator[]( const std::size_t& idx ) const ; - Json& operator=( const Json& rhs ) ; - - void Swap( Json& other ) ; - - std::string Str() const ; - int Int() const ; - double Double() const ; - bool Bool() const ; - Array AsArray() const ; - Object AsObject() const ; - - template - bool Is() const ; - - template - T As() const ; - - bool Has( const std::string& key ) const ; - bool Get( const std::string& key, Json& json ) const ; - void Add( const std::string& key, const Json& json ) ; - void Add( const Json& json ) ; - Json FindInArray( const std::string& key, const std::string& value ) const ; - bool FindInArray( const std::string& key, const std::string& value, Json& result ) const ; - - /** Expect *this is a JSON array of objects. Select all "key" values inside each - objects in the array and copies them in the output iterator \a out. - */ - template - Out Select( const std::string& key, Out out ) - { - Array a = AsArray() ; - for ( Array::iterator i = a.begin() ; i != a.end() ; ++i ) - { - Json value; - if ( i->Get( key, value ) ) - *out++ = value.As() ; - } - return out ; - } - - friend std::ostream& operator<<( std::ostream& os, const Json& json ) ; - void Write( DataStream *out ) const ; - - enum Type { null_type, bool_type, double_type, int_type, object_type, array_type, string_type } ; - - Type DataType() const ; - -private : - Json( struct json_object *json ) ; - - struct NotOwned {} ; - Json( struct json_object *json, NotOwned ) ; - - static struct json_object* InitStr( const char *str, std::size_t n ) ; - - // helper for throwing exception - template static typename Val_::Err ValueErr( const T& t ) - { - return typename Val_::Err(t); - } - -private : - struct json_object *m_json ; -} ; - -} diff --git a/libgrive/src/protocol/JsonResponse.cc b/libgrive/src/protocol/JsonResponse.cc deleted file mode 100644 index a30d0e32..00000000 --- a/libgrive/src/protocol/JsonResponse.cc +++ /dev/null @@ -1,45 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#include "JsonResponse.hh" - -#include "protocol/Json.hh" - -namespace gr { namespace http { - -JsonResponse::JsonResponse() -{ -} - -std::size_t JsonResponse::Write( const char *data, std::size_t count ) -{ - return m_resp.Write( data, count ) ; -} - -std::size_t JsonResponse::Read( char *data, std::size_t count ) -{ - return count ; -} - -Json JsonResponse::Response() const -{ - return Json::Parse( m_resp.Response() ) ; -} - -} } // end of namespace diff --git a/libgrive/src/protocol/JsonResponse.hh b/libgrive/src/protocol/JsonResponse.hh deleted file mode 100644 index 2b6322f7..00000000 --- a/libgrive/src/protocol/JsonResponse.hh +++ /dev/null @@ -1,46 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include "util/DataStream.hh" -#include "http/StringResponse.hh" - -namespace gr -{ - class Json ; -} - -namespace gr { namespace http { - -class JsonResponse : public DataStream -{ -public : - JsonResponse() ; - - std::size_t Write( const char *data, std::size_t count ) ; - std::size_t Read( char *data, std::size_t count ) ; - - Json Response() const ; - -private : - StringResponse m_resp ; -} ; - -} } // end of namespace diff --git a/libgrive/src/protocol/OAuth2.cc b/libgrive/src/protocol/OAuth2.cc index 5d2c1727..a13c8e3e 100644 --- a/libgrive/src/protocol/OAuth2.cc +++ b/libgrive/src/protocol/OAuth2.cc @@ -19,8 +19,7 @@ #include "OAuth2.hh" -#include "JsonResponse.hh" -#include "Json.hh" +#include "json/ValResponse.hh" #include "http/CurlAgent.hh" #include "http/Header.hh" @@ -61,13 +60,13 @@ void OAuth2::Auth( const std::string& auth_code ) "&redirect_uri=" + "urn:ietf:wg:oauth:2.0:oob" + "&grant_type=authorization_code" ; - http::JsonResponse resp ; + http::ValResponse resp ; http::CurlAgent http ; DisableLog dlog( log::debug ) ; http.Post( token_url, post, &resp, http::Header() ) ; - Json jresp = resp.Response() ; + Val jresp = resp.Response() ; m_access = jresp["access_token"].Str() ; m_refresh = jresp["refresh_token"].Str() ; } @@ -98,7 +97,7 @@ void OAuth2::Refresh( ) "&client_secret=" + m_client_secret + "&grant_type=refresh_token" ; - http::JsonResponse resp ; + http::ValResponse resp ; http::CurlAgent http ; DisableLog dlog( log::debug ) ; diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index 12c22c76..ca7c86ad 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -20,6 +20,8 @@ #include "Config.hh" #include "util/File.hh" +#include "json/JsonWriter.hh" +#include "json/JsonParser.hh" #include @@ -36,13 +38,13 @@ const std::string default_root_folder = "."; Config::Config( const po::variables_map& vm ) { - m_cmd.Add( "log-xml", Json(vm.count("log-xml") > 0) ) ; - m_cmd.Add( "new-rev", Json(vm.count("new-rev") > 0) ) ; - m_cmd.Add( "force", Json(vm.count("force") > 0 ) ) ; - m_cmd.Add( "path", Json(vm.count("path") > 0 + m_cmd.Add( "log-xml", Val(vm.count("log-xml") > 0) ) ; + m_cmd.Add( "new-rev", Val(vm.count("new-rev") > 0) ) ; + m_cmd.Add( "force", Val(vm.count("force") > 0 ) ) ; + m_cmd.Add( "path", Val(vm.count("path") > 0 ? vm["path"].as() : default_root_folder ) ) ; - m_cmd.Add( "dir", Json(vm.count("dir") > 0 + m_cmd.Add( "dir", Val(vm.count("dir") > 0 ? vm["dir"].as() : "" ) ) ; @@ -68,40 +70,41 @@ const fs::path Config::Filename() const void Config::Save( ) { gr::File file( m_path.string(), 0600 ) ; - m_file.Write( &file ) ; + JsonWriter wr( &file ) ; + m_file.Visit( &wr ) ; } -void Config::Set( const std::string& key, const Json& value ) +void Config::Set( const std::string& key, const Val& value ) { m_file.Add( key, value ) ; } -Json Config::Get( const std::string& key ) const +Val Config::Get( const std::string& key ) const { return m_cmd.Has(key) ? m_cmd[key] : m_file[key] ; } -Json Config::GetAll() const +Val Config::GetAll() const { - Json::Object obj = m_file.AsObject() ; - Json::Object cmd_obj = m_cmd.AsObject() ; + Val::Object obj = m_file.AsObject() ; + Val::Object cmd_obj = m_cmd.AsObject() ; - for ( Json::Object::iterator i = cmd_obj.begin() ; i != cmd_obj.end() ; ++i ) + for ( Val::Object::iterator i = cmd_obj.begin() ; i != cmd_obj.end() ; ++i ) obj[i->first] = i->second ; - return Json( obj ) ; + return Val( obj ) ; } -Json Config::Read() +Val Config::Read() { try { gr::File file(m_path) ; - return Json::Parse( &file ) ; + return ParseJson( file ) ; } catch ( Exception& e ) { - return Json() ; + return Val() ; } } diff --git a/libgrive/src/util/Config.hh b/libgrive/src/util/Config.hh index e2c0423f..9a654c56 100644 --- a/libgrive/src/util/Config.hh +++ b/libgrive/src/util/Config.hh @@ -21,7 +21,7 @@ #include "Exception.hh" #include "FileSystem.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" namespace boost { @@ -43,14 +43,14 @@ public : const fs::path Filename() const ; - void Set( const std::string& key, const Json& value ) ; - Json Get( const std::string& key ) const ; + void Set( const std::string& key, const Val& value ) ; + Val Get( const std::string& key ) const ; - Json GetAll() const ; + Val GetAll() const ; void Save() ; private : - Json Read( ) ; + Val Read( ) ; static fs::path GetPath( const fs::path& root_path ) ; private : @@ -58,10 +58,10 @@ private : fs::path m_path; //! config values loaded from config file - Json m_file ; + Val m_file ; //! config values from command line - Json m_cmd ; + Val m_cmd ; } ; } // end of namespace diff --git a/libgrive/test/btest/JsonValTest.cc b/libgrive/test/btest/JsonValTest.cc index d5087cdf..bbf2d9eb 100644 --- a/libgrive/test/btest/JsonValTest.cc +++ b/libgrive/test/btest/JsonValTest.cc @@ -38,9 +38,7 @@ BOOST_FIXTURE_TEST_SUITE( JsonValTest, F ) BOOST_AUTO_TEST_CASE( Test ) { - ValBuilder b ; - JsonParser::Parse( "{\"key\": 100 }", &b ) ; - Val json = b.Result() ; + Val json = ParseJson( "{\"key\": 100 }" ) ; BOOST_CHECK( json.Is() ) ; BOOST_CHECK_EQUAL( json["key"].As(), 100 ) ; diff --git a/libgrive/test/drive/StateTest.cc b/libgrive/test/drive/StateTest.cc index e2ed7689..167dab27 100644 --- a/libgrive/test/drive/StateTest.cc +++ b/libgrive/test/drive/StateTest.cc @@ -22,7 +22,7 @@ #include "Assert.hh" #include "drive/State.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" #include "util/log/Log.hh" #include diff --git a/libgrive/test/util/ConfigTest.cc b/libgrive/test/util/ConfigTest.cc index 48d7602c..9974dd5e 100644 --- a/libgrive/test/util/ConfigTest.cc +++ b/libgrive/test/util/ConfigTest.cc @@ -22,7 +22,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. #include "Assert.hh" #include "util/Config.hh" -#include "protocol/Json.hh" +#include "json/Val.hh" #include "util/log/Log.hh" #include diff --git a/package/fedora16/grive.spec b/package/fedora16/grive.spec index 03770b9d..3a7e7675 100644 --- a/package/fedora16/grive.spec +++ b/package/fedora16/grive.spec @@ -31,7 +31,7 @@ Source0: https://github.com/grive/%{name}/tarball/v%{version} BuildRequires: cmake BuildRequires: libstdc++-devel BuildRequires: libcurl-devel -BuildRequires: json-c-devel +BuildRequires: yajl-devel BuildRequires: expat-devel BuildRequires: openssl-devel BuildRequires: boost-devel From 770baca3fc967256886b10709aee28289bf7d758 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 16 May 2015 01:05:19 +0300 Subject: [PATCH 070/166] Reduce syscalls slightly more (os::FileCTime is still left...) --- libgrive/src/drive/Resource.cc | 4 ++-- libgrive/src/drive/State.cc | 9 +++++---- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index a9d28b92..50aa9483 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -271,7 +271,7 @@ void Resource::FromLocal( const DateTime& last_sync ) m_state = ( m_mtime > last_sync ? local_new : remote_deleted ) ; m_name = path.filename().string() ; - //m_kind = fs::is_directory(path) ? "folder" : "file" ; + m_kind = IsFolder() ? "folder" : "file" ; m_md5 = IsFolder() ? "" : crypt::MD5::Get( path ) ; } @@ -366,7 +366,7 @@ void Resource::Sync( http::Agent *http, DateTime& sync_time, const Val& options void Resource::SyncSelf( http::Agent* http, const Val& options ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync - assert( IsRoot() || http == 0 || fs::is_directory( m_parent->Path() ) ) ; + assert( IsRoot() || http == 0 || m_parent->IsFolder() ) ; assert( IsRoot() || m_parent->m_state != remote_deleted ) ; assert( IsRoot() || m_parent->m_state != local_deleted ) ; diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index c1a17e56..9326074d 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -76,21 +76,22 @@ void State::FromLocal( const fs::path& p, Resource* folder ) for ( fs::directory_iterator i( p ) ; i != fs::directory_iterator() ; ++i ) { std::string fname = i->path().filename().string() ; + fs::file_status st = fs::status(i->path()); if ( IsIgnore(fname) ) Log( "file %1% is ignored by grive", fname, log::verbose ) ; // check if it is ignored else if ( folder == m_res.Root() && m_dir != "" && fname != m_dir ) - Log( "%1% %2% is ignored", fs::is_directory(i->path()) ? "folder" : "file", fname, log::verbose ); + Log( "%1% %2% is ignored", st.type() == fs::directory_file ? "folder" : "file", fname, log::verbose ); // check for broken symblic links - else if ( !fs::exists( i->path() ) ) + else if ( st.type() == fs::file_not_found ) Log( "file %1% doesn't exist (broken link?), ignored", i->path(), log::verbose ) ; else { - bool is_dir = fs::is_directory(i->path()); + bool is_dir = st.type() == fs::directory_file; // if the Resource object of the child already exists, it should // have been so no need to do anything here Resource *c = folder->FindChild( fname ) ; @@ -101,7 +102,7 @@ void State::FromLocal( const fs::path& p, Resource* folder ) m_res.Insert( c ) ; } - c->FromLocal( m_last_sync ) ; + c->FromLocal( m_last_sync ) ; if ( is_dir ) FromLocal( *i, c ) ; From 2d8da5cab3886c26774e005618781688fecbd593 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 16 May 2015 12:35:41 +0300 Subject: [PATCH 071/166] Remove doclist API specific fields from Entry class: kind, create_link, edit_link and alt_self --- libgrive/src/drive/Drive.cc | 2 +- libgrive/src/drive/Entry.cc | 35 ++++++++++++++-------------------- libgrive/src/drive/Entry.hh | 17 +++++++---------- libgrive/src/drive/Resource.cc | 8 ++++---- libgrive/src/drive/State.cc | 21 ++++++++++---------- 5 files changed, 37 insertions(+), 46 deletions(-) diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index c5e4eef3..bdc5c779 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -108,7 +108,7 @@ void Drive::SyncFolders( ) for ( Feed::iterator i = feed.begin() ; i != feed.end() ; ++i ) { Entry e( *i ) ; - if ( e.Kind() == "folder" ) + if ( e.IsDir() ) { if ( e.ParentHrefs().size() != 1 ) Log( "folder \"%1%\" has multiple parents, ignored", e.Title(), log::verbose ) ; diff --git a/libgrive/src/drive/Entry.cc b/libgrive/src/drive/Entry.cc index 506295e1..7d54fbf5 100644 --- a/libgrive/src/drive/Entry.cc +++ b/libgrive/src/drive/Entry.cc @@ -34,10 +34,9 @@ namespace gr { namespace v1 { /// construct an entry for the root folder Entry::Entry( ) : m_title ( "." ), - m_kind ( "folder" ), + m_is_dir ( true ), m_resource_id ( "folder:root" ), m_self_href ( root_href ), - m_create_link ( root_create ), m_change_stamp ( -1 ), m_is_removed ( false ) { @@ -58,18 +57,22 @@ void Entry::Update( const xml::Node& n ) m_filename = n["docs:suggestedFilename"] ; m_content_src = n["content"]["@src"] ; m_self_href = n["link"].Find( "@rel", "self" )["@href"] ; - m_alt_self = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#alt-self" )["@href"] ; m_mtime = DateTime( n["updated"] ) ; m_resource_id = n["gd:resourceId"] ; m_md5 = n["docs:md5Checksum"] ; - m_kind = n["category"].Find( "@scheme", "http://schemas.google.com/g/2005#kind" )["@label"] ; - m_edit_link = n["link"].Find( "@rel", "http://schemas.google.com/g/2005#resumable-edit-media")["@href"] ; - m_create_link = n["link"].Find( "@rel", "http://schemas.google.com/g/2005#resumable-create-media")["@href"] ; + m_is_dir = n["category"].Find( "@scheme", "http://schemas.google.com/g/2005#kind" )["@label"] == "folder" ; + m_is_editable = !n["link"].Find( "@rel", m_is_dir + ? "http://schemas.google.com/g/2005#resumable-create-media" : "http://schemas.google.com/g/2005#resumable-edit-media" ) + ["@href"].empty() ; // changestamp only appear in change feed entries xml::NodeSet cs = n["docs:changestamp"]["@value"] ; m_change_stamp = cs.empty() ? -1 : std::atoi( cs.front().Value().c_str() ) ; + if ( m_change_stamp != -1 ) + { + m_self_href = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#alt-self" )["@href"] ; + } m_parent_hrefs.clear( ) ; xml::NodeSet parents = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#parent" ) ; @@ -97,9 +100,9 @@ std::string Entry::Filename() const return m_filename ; } -std::string Entry::Kind() const +bool Entry::IsDir() const { - return m_kind ; + return m_is_dir ; } std::string Entry::MD5() const @@ -117,11 +120,6 @@ std::string Entry::SelfHref() const return m_self_href ; } -std::string Entry::AltSelf() const -{ - return m_alt_self ; -} - std::string Entry::ParentHref() const { return m_parent_hrefs.empty() ? "" : m_parent_hrefs.front() ; @@ -142,14 +140,9 @@ std::string Entry::ContentSrc() const return m_content_src ; } -std::string Entry::EditLink() const -{ - return m_edit_link ; -} - -std::string Entry::CreateLink() const +bool Entry::IsEditable() const { - return m_create_link ; + return m_is_editable ; } long Entry::ChangeStamp() const @@ -169,7 +162,7 @@ bool Entry::IsRemoved() const std::string Entry::Name() const { - return (m_kind == "file" || m_kind == "pdf") ? m_filename : m_title ; + return !m_filename.empty() ? m_filename : m_title ; } } } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Entry.hh b/libgrive/src/drive/Entry.hh index a059b4f0..2d1c5fc5 100644 --- a/libgrive/src/drive/Entry.hh +++ b/libgrive/src/drive/Entry.hh @@ -49,7 +49,7 @@ public : std::string Title() const ; std::string Filename() const ; - std::string Kind() const ; + bool IsDir() const ; std::string MD5() const ; DateTime MTime() const ; @@ -59,11 +59,9 @@ public : std::string ETag() const ; std::string SelfHref() const ; - std::string AltSelf() const ; std::string ParentHref() const ; std::string ContentSrc() const ; - std::string EditLink() const ; - std::string CreateLink() const ; + bool IsEditable() const ; long ChangeStamp() const ; bool IsChange() const ; @@ -76,21 +74,20 @@ public : private : std::string m_title ; std::string m_filename ; - std::string m_kind ; + bool m_is_dir ; std::string m_md5 ; std::string m_etag ; std::string m_resource_id ; std::vector m_parent_hrefs ; - + std::string m_self_href ; - std::string m_alt_self ; std::string m_content_src ; - std::string m_edit_link ; - std::string m_create_link ; + + bool m_is_editable ; long m_change_stamp ; - + DateTime m_mtime ; bool m_is_removed ; } ; diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 50aa9483..42862e03 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -95,7 +95,7 @@ void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_sync { fs::path path = Path() ; - if ( remote.CreateLink().empty() ) + if ( !remote.IsEditable() ) Log( "folder %1% is read-only", path, log::verbose ) ; // already sync @@ -143,7 +143,7 @@ void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_sync void Resource::FromRemote( const Entry& remote, const DateTime& last_sync ) { // sync folder - if ( remote.Kind() == "folder" && IsFolder() ) + if ( remote.IsDir() && IsFolder() ) FromRemoteFolder( remote, last_sync ) ; else FromRemoteFile( remote, last_sync ) ; @@ -166,8 +166,8 @@ void Resource::AssignIDs( const Entry& remote ) { m_id = remote.ResourceID() ; m_href = remote.SelfHref() ; - m_edit = remote.EditLink() ; - m_create = remote.CreateLink() ; + m_edit = remote.IsEditable() ? feed_base + "/" + remote.ResourceID() : ""; + m_create = remote.IsEditable() && remote.IsDir() ? root_create + (remote.ResourceID() == "folder:root" ? "" : "/" + remote.ResourceID() + "/contents") : ""; m_content = remote.ContentSrc() ; m_etag = remote.ETag() ; } diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index 9326074d..e464b437 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -112,24 +112,25 @@ void State::FromLocal( const fs::path& p, Resource* folder ) void State::FromRemote( const Entry& e ) { - std::string fn = e.Filename() ; + std::string fn = e.Filename() ; + std::string k = e.IsDir() ? "folder" : "file"; if ( IsIgnore( e.Name() ) ) - Log( "%1% %2% is ignored by grive", e.Kind(), e.Name(), log::verbose ) ; + Log( "%1% %2% is ignored by grive", k, e.Name(), log::verbose ) ; // check if it is ignored else if ( e.ParentHref() == m_res.Root()->SelfHref() && m_dir != "" && e.Name() != m_dir ) - Log( "%1% %2% is ignored", e.Kind(), e.Name(), log::verbose ); + Log( "%1% %2% is ignored", k, e.Name(), log::verbose ); // common checkings - else if ( e.Kind() != "folder" && (fn.empty() || e.ContentSrc().empty()) ) - Log( "%1% \"%2%\" is a google document, ignored", e.Kind(), e.Name(), log::verbose ) ; + else if ( !e.IsDir() && (fn.empty() || e.ContentSrc().empty()) ) + Log( "%1% \"%2%\" is a google document, ignored", k, e.Name(), log::verbose ) ; else if ( fn.find('/') != fn.npos ) - Log( "%1% \"%2%\" contains a slash in its name, ignored", e.Kind(), e.Name(), log::verbose ) ; + Log( "%1% \"%2%\" contains a slash in its name, ignored", k, e.Name(), log::verbose ) ; else if ( !e.IsChange() && e.ParentHrefs().size() != 1 ) - Log( "%1% \"%2%\" has multiple parents, ignored", e.Kind(), e.Name(), log::verbose ) ; + Log( "%1% \"%2%\" has multiple parents, ignored", k, e.Name(), log::verbose ) ; else if ( e.IsChange() ) FromChange( e ) ; @@ -176,7 +177,7 @@ void State::FromChange( const Entry& e ) // entries in the change feed is always treated as newer in remote, // so we override the last sync time to 0 - if ( Resource *res = m_res.FindByHref( e.AltSelf() ) ) + if ( Resource *res = m_res.FindByHref( e.SelfHref() ) ) m_res.Update( res, e, DateTime() ) ; } @@ -205,10 +206,10 @@ bool State::Update( const Entry& e ) // folder entry exist in google drive, but not local. we should create // the directory - else if ( e.Kind() == "folder" || !e.Filename().empty() ) + else if ( e.IsDir() || !e.Filename().empty() ) { // first create a dummy resource and update it later - child = new Resource( name, e.Kind() ) ; + child = new Resource( name, e.IsDir() ? "folder" : "file" ) ; parent->AddChild( child ) ; m_res.Insert( child ) ; From b0255d9699f9932e519e3d334eb76a2a4bf17195 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 16 May 2015 19:38:43 +0300 Subject: [PATCH 072/166] Add Entry constructor for new Drive REST API --- libgrive/src/drive/Entry.cc | 52 ++++++++++++++++++++++++++++++++++++- libgrive/src/drive/Entry.hh | 6 ++++- libgrive/src/json/Val.cc | 11 +++++--- libgrive/src/json/Val.hh | 4 +-- 4 files changed, 66 insertions(+), 7 deletions(-) diff --git a/libgrive/src/drive/Entry.cc b/libgrive/src/drive/Entry.cc index 7d54fbf5..bbd24ce5 100644 --- a/libgrive/src/drive/Entry.cc +++ b/libgrive/src/drive/Entry.cc @@ -26,6 +26,9 @@ #include "xml/Node.hh" #include "xml/NodeSet.hh" +#include "json/Val.hh" +#include "drive2/CommonUri.hh" + #include #include @@ -42,7 +45,7 @@ Entry::Entry( ) : { } -/// construct an entry for remote +/// construct an entry for remote - Doclist API v3 Entry::Entry( const xml::Node& n ) : m_change_stamp( -1 ), m_is_removed( false ) @@ -50,6 +53,14 @@ Entry::Entry( const xml::Node& n ) : Update( n ) ; } +/// construct an entry for remote, from "file" JSON object - Drive REST API +Entry::Entry( const Val& item ) : + m_change_stamp( -1 ), + m_is_removed( false ) +{ + Update( item ) ; +} + void Entry::Update( const xml::Node& n ) { m_title = n["title"] ; @@ -85,6 +96,45 @@ void Entry::Update( const xml::Node& n ) m_is_removed = !n["gd:deleted"].empty() || !n["docs:removed"].empty() ; } +void Entry::Update( const Val& item ) +{ + bool is_chg = item["kind"].Str() == "drive#change"; + + // changestamp only appears in change feed entries + m_change_stamp = is_chg ? item["id"].Int() : -1 ; + m_is_removed = is_chg && item["deleted"].Bool() ; + + const Val& file = is_chg && !m_is_removed ? item["file"] : item; + + if ( m_is_removed ) + { + m_resource_id = item["fileId"]; + } + else + { + m_title = file["title"] ; + m_etag = file["etag"] ; + m_filename = file["originalFilename"] ; + m_content_src = file["downloadUrl"] ; + m_self_href = file["selfLink"] ; + m_mtime = DateTime( file["modificationDate"] ) ; + + m_resource_id = file["id"]; // file#id ? + m_md5 = file["md5Checksum"] ; + m_is_dir = file["mimeType"].Str() == v2::mime_types::folder ; + m_is_editable = file["editable"].Bool() ; + + m_parent_hrefs.clear( ) ; + + Val::Array parents = file["parents"].AsArray() ; + for ( Val::Array::iterator i = parents.begin() ; i != parents.end() ; ++i ) + m_parent_hrefs.push_back( (*i)["parentLink"] ) ; // maybe .id? + + // convert to lower case for easy comparison + std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; + } +} + const std::vector& Entry::ParentHrefs() const { return m_parent_hrefs ; diff --git a/libgrive/src/drive/Entry.hh b/libgrive/src/drive/Entry.hh index 2d1c5fc5..dc4a5dad 100644 --- a/libgrive/src/drive/Entry.hh +++ b/libgrive/src/drive/Entry.hh @@ -28,6 +28,8 @@ namespace gr { +class Val ; + namespace xml { class Node ; @@ -46,6 +48,7 @@ class Entry public : Entry( ) ; explicit Entry( const xml::Node& n ) ; + explicit Entry( const Val& item ) ; std::string Title() const ; std::string Filename() const ; @@ -69,9 +72,10 @@ public : const std::vector& ParentHrefs() const ; +private : void Update( const xml::Node& entry ) ; + void Update( const Val& item ) ; -private : std::string m_title ; std::string m_filename ; bool m_is_dir ; diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index 79c0b5e1..31944ffe 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -107,17 +107,22 @@ std::string Val::Str() const return As() ; } -int Val::Int() const +Val::operator std::string() const +{ + return As() ; +} + +int Val::Int() const { return static_cast(As()) ; } -double Val::Double() const +double Val::Double() const { return As() ; } -bool Val::Bool() const +bool Val::Bool() const { return As() ; } diff --git a/libgrive/src/json/Val.hh b/libgrive/src/json/Val.hh index 25478860..e5673b25 100644 --- a/libgrive/src/json/Val.hh +++ b/libgrive/src/json/Val.hh @@ -81,6 +81,8 @@ public : return Assign(t) ; } + operator std::string() const ; + template const T& As() const ; @@ -111,8 +113,6 @@ public : // shortcuts for array (and array of objects) void Add( const Val& json ) ; - Val FindInArray( const std::string& key, const std::string& value ) const ; - bool FindInArray( const std::string& key, const std::string& value, Val& result ) const ; std::vector Select( const std::string& key ) const ; From f41e4f6dd38e6b3c8d75782c55126b46f34555b6 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 16 May 2015 21:10:38 +0300 Subject: [PATCH 073/166] Split Entry to v1::Entry1 and v2::Entry2 --- libgrive/CMakeLists.txt | 1 + libgrive/src/base/Entry.cc | 123 +++++++++++++++ libgrive/src/{drive => base}/Entry.hh | 18 +-- libgrive/src/drive/Drive.cc | 2 +- libgrive/src/drive/Drive.hh | 4 +- libgrive/src/drive/Entry.cc | 218 -------------------------- libgrive/src/drive/Entry1.cc | 81 ++++++++++ libgrive/src/drive/Entry1.hh | 42 +++++ libgrive/src/drive/Feed.cc | 4 +- libgrive/src/drive/Feed.hh | 2 +- libgrive/src/drive/Resource.cc | 8 +- libgrive/src/drive/Resource.hh | 4 +- libgrive/src/drive/State.cc | 2 +- libgrive/src/drive/State.hh | 3 +- libgrive/src/drive2/Entry2.cc | 79 ++++++++++ libgrive/src/drive2/Entry2.hh | 38 +++++ 16 files changed, 382 insertions(+), 247 deletions(-) create mode 100644 libgrive/src/base/Entry.cc rename libgrive/src/{drive => base}/Entry.hh (88%) delete mode 100644 libgrive/src/drive/Entry.cc create mode 100644 libgrive/src/drive/Entry1.cc create mode 100644 libgrive/src/drive/Entry1.hh create mode 100644 libgrive/src/drive2/Entry2.cc create mode 100644 libgrive/src/drive2/Entry2.hh diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index fd610532..5a6558e2 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -64,6 +64,7 @@ file (GLOB XML_HEADERS ) file (GLOB LIBGRIVE_SRC + src/base/*.cc src/drive/*.cc src/drive2/*.cc src/http/*.cc diff --git a/libgrive/src/base/Entry.cc b/libgrive/src/base/Entry.cc new file mode 100644 index 00000000..ed25a101 --- /dev/null +++ b/libgrive/src/base/Entry.cc @@ -0,0 +1,123 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "Entry.hh" + +#include "util/Crypt.hh" +#include "util/log/Log.hh" +#include "util/OS.hh" +#include "xml/Node.hh" +#include "xml/NodeSet.hh" + +#include +#include + +namespace gr { + +/// construct an entry for the root folder +Entry::Entry( ) : + m_title ( "." ), + m_is_dir ( true ), + m_resource_id ( "folder:root" ), + m_change_stamp ( -1 ), + m_is_removed ( false ) +{ +} + +const std::vector& Entry::ParentHrefs() const +{ + return m_parent_hrefs ; +} + +std::string Entry::Title() const +{ + return m_title ; +} + +std::string Entry::Filename() const +{ + return m_filename ; +} + +bool Entry::IsDir() const +{ + return m_is_dir ; +} + +std::string Entry::MD5() const +{ + return m_md5 ; +} + +DateTime Entry::MTime() const +{ + return m_mtime ; +} + +std::string Entry::SelfHref() const +{ + return m_self_href ; +} + +std::string Entry::ParentHref() const +{ + return m_parent_hrefs.empty() ? "" : m_parent_hrefs.front() ; +} + +std::string Entry::ResourceID() const +{ + return m_resource_id ; +} + +std::string Entry::ETag() const +{ + return m_etag ; +} + +std::string Entry::ContentSrc() const +{ + return m_content_src ; +} + +bool Entry::IsEditable() const +{ + return m_is_editable ; +} + +long Entry::ChangeStamp() const +{ + return m_change_stamp ; +} + +bool Entry::IsChange() const +{ + return m_change_stamp != -1 ; +} + +bool Entry::IsRemoved() const +{ + return m_is_removed ; +} + +std::string Entry::Name() const +{ + return !m_filename.empty() ? m_filename : m_title ; +} + +} // end of namespace gr diff --git a/libgrive/src/drive/Entry.hh b/libgrive/src/base/Entry.hh similarity index 88% rename from libgrive/src/drive/Entry.hh rename to libgrive/src/base/Entry.hh index dc4a5dad..6472e892 100644 --- a/libgrive/src/drive/Entry.hh +++ b/libgrive/src/base/Entry.hh @@ -28,15 +28,6 @@ namespace gr { -class Val ; - -namespace xml -{ - class Node ; -} - -namespace v1 { - /*! \brief corresponds to an "entry" in the resource feed This class is decodes an entry in the resource feed. It will stored the properties like @@ -47,8 +38,6 @@ class Entry { public : Entry( ) ; - explicit Entry( const xml::Node& n ) ; - explicit Entry( const Val& item ) ; std::string Title() const ; std::string Filename() const ; @@ -72,10 +61,7 @@ public : const std::vector& ParentHrefs() const ; -private : - void Update( const xml::Node& entry ) ; - void Update( const Val& item ) ; - +protected : std::string m_title ; std::string m_filename ; bool m_is_dir ; @@ -96,4 +82,4 @@ private : bool m_is_removed ; } ; -} } // end of namespace gr::v1 +} // end of namespace gr diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index bdc5c779..a8b99b62 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -20,7 +20,7 @@ #include "Drive.hh" #include "CommonUri.hh" -#include "Entry.hh" +#include "base/Entry.hh" #include "Feed.hh" #include "http/Agent.hh" diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 1f1e4758..57c24f9b 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -35,10 +35,10 @@ namespace http class Agent ; } -namespace v1 { - class Entry ; +namespace v1 { + class Drive { public : diff --git a/libgrive/src/drive/Entry.cc b/libgrive/src/drive/Entry.cc deleted file mode 100644 index bbd24ce5..00000000 --- a/libgrive/src/drive/Entry.cc +++ /dev/null @@ -1,218 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#include "Entry.hh" -#include "CommonUri.hh" - -#include "util/Crypt.hh" -#include "util/log/Log.hh" -#include "util/OS.hh" -#include "xml/Node.hh" -#include "xml/NodeSet.hh" - -#include "json/Val.hh" -#include "drive2/CommonUri.hh" - -#include -#include - -namespace gr { namespace v1 { - -/// construct an entry for the root folder -Entry::Entry( ) : - m_title ( "." ), - m_is_dir ( true ), - m_resource_id ( "folder:root" ), - m_self_href ( root_href ), - m_change_stamp ( -1 ), - m_is_removed ( false ) -{ -} - -/// construct an entry for remote - Doclist API v3 -Entry::Entry( const xml::Node& n ) : - m_change_stamp( -1 ), - m_is_removed( false ) -{ - Update( n ) ; -} - -/// construct an entry for remote, from "file" JSON object - Drive REST API -Entry::Entry( const Val& item ) : - m_change_stamp( -1 ), - m_is_removed( false ) -{ - Update( item ) ; -} - -void Entry::Update( const xml::Node& n ) -{ - m_title = n["title"] ; - m_etag = n["@gd:etag"] ; - m_filename = n["docs:suggestedFilename"] ; - m_content_src = n["content"]["@src"] ; - m_self_href = n["link"].Find( "@rel", "self" )["@href"] ; - m_mtime = DateTime( n["updated"] ) ; - - m_resource_id = n["gd:resourceId"] ; - m_md5 = n["docs:md5Checksum"] ; - m_is_dir = n["category"].Find( "@scheme", "http://schemas.google.com/g/2005#kind" )["@label"] == "folder" ; - m_is_editable = !n["link"].Find( "@rel", m_is_dir - ? "http://schemas.google.com/g/2005#resumable-create-media" : "http://schemas.google.com/g/2005#resumable-edit-media" ) - ["@href"].empty() ; - - // changestamp only appear in change feed entries - xml::NodeSet cs = n["docs:changestamp"]["@value"] ; - m_change_stamp = cs.empty() ? -1 : std::atoi( cs.front().Value().c_str() ) ; - if ( m_change_stamp != -1 ) - { - m_self_href = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#alt-self" )["@href"] ; - } - - m_parent_hrefs.clear( ) ; - xml::NodeSet parents = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#parent" ) ; - for ( xml::NodeSet::iterator i = parents.begin() ; i != parents.end() ; ++i ) - m_parent_hrefs.push_back( (*i)["@href"] ) ; - - // convert to lower case for easy comparison - std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; - - m_is_removed = !n["gd:deleted"].empty() || !n["docs:removed"].empty() ; -} - -void Entry::Update( const Val& item ) -{ - bool is_chg = item["kind"].Str() == "drive#change"; - - // changestamp only appears in change feed entries - m_change_stamp = is_chg ? item["id"].Int() : -1 ; - m_is_removed = is_chg && item["deleted"].Bool() ; - - const Val& file = is_chg && !m_is_removed ? item["file"] : item; - - if ( m_is_removed ) - { - m_resource_id = item["fileId"]; - } - else - { - m_title = file["title"] ; - m_etag = file["etag"] ; - m_filename = file["originalFilename"] ; - m_content_src = file["downloadUrl"] ; - m_self_href = file["selfLink"] ; - m_mtime = DateTime( file["modificationDate"] ) ; - - m_resource_id = file["id"]; // file#id ? - m_md5 = file["md5Checksum"] ; - m_is_dir = file["mimeType"].Str() == v2::mime_types::folder ; - m_is_editable = file["editable"].Bool() ; - - m_parent_hrefs.clear( ) ; - - Val::Array parents = file["parents"].AsArray() ; - for ( Val::Array::iterator i = parents.begin() ; i != parents.end() ; ++i ) - m_parent_hrefs.push_back( (*i)["parentLink"] ) ; // maybe .id? - - // convert to lower case for easy comparison - std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; - } -} - -const std::vector& Entry::ParentHrefs() const -{ - return m_parent_hrefs ; -} - -std::string Entry::Title() const -{ - return m_title ; -} - -std::string Entry::Filename() const -{ - return m_filename ; -} - -bool Entry::IsDir() const -{ - return m_is_dir ; -} - -std::string Entry::MD5() const -{ - return m_md5 ; -} - -DateTime Entry::MTime() const -{ - return m_mtime ; -} - -std::string Entry::SelfHref() const -{ - return m_self_href ; -} - -std::string Entry::ParentHref() const -{ - return m_parent_hrefs.empty() ? "" : m_parent_hrefs.front() ; -} - -std::string Entry::ResourceID() const -{ - return m_resource_id ; -} - -std::string Entry::ETag() const -{ - return m_etag ; -} - -std::string Entry::ContentSrc() const -{ - return m_content_src ; -} - -bool Entry::IsEditable() const -{ - return m_is_editable ; -} - -long Entry::ChangeStamp() const -{ - return m_change_stamp ; -} - -bool Entry::IsChange() const -{ - return m_change_stamp != -1 ; -} - -bool Entry::IsRemoved() const -{ - return m_is_removed ; -} - -std::string Entry::Name() const -{ - return !m_filename.empty() ? m_filename : m_title ; -} - -} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Entry1.cc b/libgrive/src/drive/Entry1.cc new file mode 100644 index 00000000..2d1d61a8 --- /dev/null +++ b/libgrive/src/drive/Entry1.cc @@ -0,0 +1,81 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "Entry1.hh" +#include "CommonUri.hh" + +#include "util/Crypt.hh" +#include "util/log/Log.hh" +#include "util/OS.hh" +#include "xml/Node.hh" +#include "xml/NodeSet.hh" + +#include +#include + +namespace gr { namespace v1 { + +Entry1::Entry1(): + Entry () +{ + m_self_href = root_href; +} + +/// construct an entry for remote - Doclist API v3 +Entry1::Entry1( const xml::Node& n ) +{ + Update( n ) ; +} + +void Entry1::Update( const xml::Node& n ) +{ + m_title = n["title"] ; + m_etag = n["@gd:etag"] ; + m_filename = n["docs:suggestedFilename"] ; + m_content_src = n["content"]["@src"] ; + m_self_href = n["link"].Find( "@rel", "self" )["@href"] ; + m_mtime = DateTime( n["updated"] ) ; + + m_resource_id = n["gd:resourceId"] ; + m_md5 = n["docs:md5Checksum"] ; + m_is_dir = n["category"].Find( "@scheme", "http://schemas.google.com/g/2005#kind" )["@label"] == "folder" ; + m_is_editable = !n["link"].Find( "@rel", m_is_dir + ? "http://schemas.google.com/g/2005#resumable-create-media" : "http://schemas.google.com/g/2005#resumable-edit-media" ) + ["@href"].empty() ; + + // changestamp only appear in change feed entries + xml::NodeSet cs = n["docs:changestamp"]["@value"] ; + m_change_stamp = cs.empty() ? -1 : std::atoi( cs.front().Value().c_str() ) ; + if ( m_change_stamp != -1 ) + { + m_self_href = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#alt-self" )["@href"] ; + } + + m_parent_hrefs.clear( ) ; + xml::NodeSet parents = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#parent" ) ; + for ( xml::NodeSet::iterator i = parents.begin() ; i != parents.end() ; ++i ) + m_parent_hrefs.push_back( (*i)["@href"] ) ; + + // convert to lower case for easy comparison + std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; + + m_is_removed = !n["gd:deleted"].empty() || !n["docs:removed"].empty() ; +} + +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Entry1.hh b/libgrive/src/drive/Entry1.hh new file mode 100644 index 00000000..b606f8af --- /dev/null +++ b/libgrive/src/drive/Entry1.hh @@ -0,0 +1,42 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "base/Entry.hh" + +namespace gr { + +namespace xml +{ + class Node ; +} + +namespace v1 { + +class Entry1: public Entry +{ +public : + Entry1( ) ; + explicit Entry1( const xml::Node& n ) ; +private : + void Update( const xml::Node& entry ) ; +} ; + +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed.cc index b227e520..f57545d0 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed.cc @@ -19,6 +19,8 @@ #include "Feed.hh" +#include "Entry1.hh" + #include "http/Agent.hh" #include "http/Header.hh" #include "http/ResponseLog.hh" @@ -107,7 +109,7 @@ Feed::iterator::iterator( xml::Node::iterator i ) Feed::iterator::reference Feed::iterator::dereference() const { - return Entry( *base_reference() ) ; + return Entry1( *base_reference() ) ; } void Feed::EnableLog( const std::string& prefix, const std::string& suffix ) diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/drive/Feed.hh index 92e62345..ce85435d 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/drive/Feed.hh @@ -19,7 +19,7 @@ #pragma once -#include "Entry.hh" +#include "base/Entry.hh" #include "xml/Node.hh" #include "xml/NodeSet.hh" diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/drive/Resource.cc index 42862e03..53ba6a45 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/drive/Resource.cc @@ -19,7 +19,7 @@ #include "Resource.hh" #include "CommonUri.hh" -#include "Entry.hh" +#include "Entry1.hh" #include "http/Agent.hh" #include "http/Download.hh" @@ -472,7 +472,7 @@ void Resource::DeleteRemote( http::Agent *http ) // doesn't know why, but an update before deleting seems to work always http::XmlResponse xml ; http->Get( m_href, &xml, hdr ) ; - AssignIDs( Entry( xml.Response() ) ) ; + AssignIDs( Entry1( xml.Response() ) ) ; http->Custom( "DELETE", m_href, &str, hdr ) ; } @@ -542,7 +542,7 @@ bool Resource::Create( http::Agent* http ) http::XmlResponse xml ; // http::ResponseLog log( "create", ".xml", &xml ) ; http->Post( uri, meta, &xml, hdr ) ; - AssignIDs( Entry( xml.Response() ) ) ; + AssignIDs( Entry1( xml.Response() ) ) ; return true ; } @@ -637,7 +637,7 @@ bool Resource::Upload( if ( retrying ) Log( "upload succeeded on retry", log::warning ); - Entry responseEntry = Entry( xml.Response() ); + Entry1 responseEntry = Entry1( xml.Response() ); AssignIDs( responseEntry ) ; m_mtime = responseEntry.MTime(); break; diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/drive/Resource.hh index 57f9262c..62cea693 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/drive/Resource.hh @@ -36,10 +36,10 @@ namespace http class Val ; -namespace v1 { - class Entry ; +namespace v1 { + /*! \brief A resource can be a file or a folder in the google drive The google drive contains a number of resources, which is represented by this class. diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index e464b437..d172fcc2 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -19,7 +19,7 @@ #include "State.hh" -#include "Entry.hh" +#include "base/Entry.hh" #include "Resource.hh" #include "CommonUri.hh" diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index 50492519..f2acfb98 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -35,10 +35,11 @@ namespace http class Val ; +class Entry ; + namespace v1 { class Resource ; -class Entry ; class State { diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc new file mode 100644 index 00000000..14795231 --- /dev/null +++ b/libgrive/src/drive2/Entry2.cc @@ -0,0 +1,79 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "Entry2.hh" +#include "CommonUri.hh" + +#include "util/Crypt.hh" +#include "util/log/Log.hh" +#include "util/OS.hh" + +#include "json/Val.hh" + +#include +#include + +namespace gr { namespace v2 { + +/// construct an entry for remote, from "file" JSON object - Drive REST API +Entry2::Entry2( const Val& item ) +{ + Update( item ) ; +} + +void Entry2::Update( const Val& item ) +{ + bool is_chg = item["kind"].Str() == "drive#change"; + + // changestamp only appears in change feed entries + m_change_stamp = is_chg ? item["id"].Int() : -1 ; + m_is_removed = is_chg && item["deleted"].Bool() ; + + const Val& file = is_chg && !m_is_removed ? item["file"] : item; + + if ( m_is_removed ) + { + m_resource_id = item["fileId"]; + } + else + { + m_title = file["title"] ; + m_etag = file["etag"] ; + m_filename = file["originalFilename"] ; + m_content_src = file["downloadUrl"] ; + m_self_href = file["selfLink"] ; + m_mtime = DateTime( file["modificationDate"] ) ; + + m_resource_id = file["id"]; // file#id ? + m_md5 = file["md5Checksum"] ; + m_is_dir = file["mimeType"].Str() == v2::mime_types::folder ; + m_is_editable = file["editable"].Bool() ; + + m_parent_hrefs.clear( ) ; + + Val::Array parents = file["parents"].AsArray() ; + for ( Val::Array::iterator i = parents.begin() ; i != parents.end() ; ++i ) + m_parent_hrefs.push_back( (*i)["parentLink"] ) ; // maybe .id? + + // convert to lower case for easy comparison + std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; + } +} + +} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Entry2.hh b/libgrive/src/drive2/Entry2.hh new file mode 100644 index 00000000..624df30a --- /dev/null +++ b/libgrive/src/drive2/Entry2.hh @@ -0,0 +1,38 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "base/Entry.hh" + +namespace gr { + +class Val ; + +namespace v2 { + +class Entry2: public Entry +{ +public : + explicit Entry2( const Val& item ) ; +private : + void Update( const Val& item ) ; +} ; + +} } // end of namespace gr::v2 From 6a15dd09a5d80c9b079ebb32af7082aeb014b442 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 02:00:00 +0300 Subject: [PATCH 074/166] Split Resource to global (base/Resource) and API-specific ({drive,drive2}/Syncer) parts --- libgrive/src/{drive => base}/Resource.cc | 289 +++++------------------ libgrive/src/{drive => base}/Resource.hh | 107 ++++----- libgrive/src/base/Syncer.cc | 42 ++++ libgrive/src/base/Syncer.hh | 62 +++++ libgrive/src/drive/Drive.cc | 7 +- libgrive/src/drive/ResourceTree.hh | 2 +- libgrive/src/drive/State.cc | 8 +- libgrive/src/drive/State.hh | 11 +- libgrive/src/drive/Syncer1.cc | 255 ++++++++++++++++++++ libgrive/src/drive/Syncer1.hh | 46 ++++ libgrive/src/drive2/Drive.cc | 164 ------------- libgrive/src/drive2/Drive.hh | 89 ------- libgrive/src/drive2/Feed.cc | 1 + libgrive/src/drive2/Resource.cc | 100 -------- libgrive/src/drive2/Resource.hh | 62 ----- 15 files changed, 526 insertions(+), 719 deletions(-) rename libgrive/src/{drive => base}/Resource.cc (63%) rename libgrive/src/{drive => base}/Resource.hh (86%) create mode 100644 libgrive/src/base/Syncer.cc create mode 100644 libgrive/src/base/Syncer.hh create mode 100644 libgrive/src/drive/Syncer1.cc create mode 100644 libgrive/src/drive/Syncer1.hh delete mode 100644 libgrive/src/drive2/Drive.cc delete mode 100644 libgrive/src/drive2/Drive.hh delete mode 100644 libgrive/src/drive2/Resource.cc delete mode 100644 libgrive/src/drive2/Resource.hh diff --git a/libgrive/src/drive/Resource.cc b/libgrive/src/base/Resource.cc similarity index 63% rename from libgrive/src/drive/Resource.cc rename to libgrive/src/base/Resource.cc index 53ba6a45..ef995ab6 100644 --- a/libgrive/src/drive/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -18,55 +18,33 @@ */ #include "Resource.hh" -#include "CommonUri.hh" -#include "Entry1.hh" - -#include "http/Agent.hh" -#include "http/Download.hh" -#include "http/Header.hh" -// #include "http/ResponseLog.hh" -#include "http/StringResponse.hh" -#include "http/XmlResponse.hh" +#include "Entry.hh" +#include "Syncer.hh" + #include "json/Val.hh" #include "util/CArray.hh" #include "util/Crypt.hh" #include "util/log/Log.hh" #include "util/OS.hh" #include "util/File.hh" -#include "xml/Node.hh" -#include "xml/NodeSet.hh" -#include "xml/String.hh" -#include "xml/TreeBuilder.hh" #include -#include #include // for debugging #include -namespace gr { namespace v1 { - -// hard coded XML file -const std::string xml_meta = - "\n" - "" - "" - "%2%" - "" ; - +namespace gr { /// default constructor creates the root folder Resource::Resource(const fs::path& root_folder) : m_name ( root_folder.string() ), m_kind ( "folder" ), m_id ( "folder:root" ), - m_href ( root_href ), - m_create ( root_create ), m_parent ( 0 ), - m_state ( sync ) + m_state ( sync ), + m_is_editable( true ) { } @@ -74,7 +52,8 @@ Resource::Resource( const std::string& name, const std::string& kind ) : m_name ( name ), m_kind ( kind ), m_parent ( 0 ), - m_state ( unknown ) + m_state ( unknown ), + m_is_editable( true ) { } @@ -166,9 +145,8 @@ void Resource::AssignIDs( const Entry& remote ) { m_id = remote.ResourceID() ; m_href = remote.SelfHref() ; - m_edit = remote.IsEditable() ? feed_base + "/" + remote.ResourceID() : ""; - m_create = remote.IsEditable() && remote.IsDir() ? root_create + (remote.ResourceID() == "folder:root" ? "" : "/" + remote.ResourceID() + "/contents") : ""; m_content = remote.ContentSrc() ; + m_is_editable = remote.IsEditable() ; m_etag = remote.ETag() ; } } @@ -283,16 +261,41 @@ std::string Resource::SelfHref() const return m_href ; } +std::string Resource::ContentSrc() const +{ + return m_content ; +} + +std::string Resource::ETag() const +{ + return m_etag ; +} + std::string Resource::Name() const { return m_name ; } +std::string Resource::Kind() const +{ + return m_kind ; +} + +DateTime Resource::MTime() const +{ + return m_mtime ; +} + std::string Resource::ResourceID() const { return m_id ; } +Resource::State Resource::GetState() const +{ + return m_state ; +} + const Resource* Resource::Parent() const { assert( m_parent == 0 || m_parent->IsFolder() ) ; @@ -320,6 +323,11 @@ bool Resource::IsFolder() const return m_kind == "folder" ; } +bool Resource::IsEditable() const +{ + return m_is_editable ; +} + fs::path Resource::Path() const { assert( m_parent != this ) ; @@ -331,7 +339,7 @@ fs::path Resource::Path() const bool Resource::IsInRootTree() const { assert( m_parent == 0 || m_parent->IsFolder() ) ; - return m_parent == 0 ? (SelfHref() == root_href) : m_parent->IsInRootTree() ; + return m_parent == 0 ? SelfHref().empty() : m_parent->IsInRootTree() ; } Resource* Resource::FindChild( const std::string& name ) @@ -346,12 +354,12 @@ Resource* Resource::FindChild( const std::string& name ) } // try to change the state to "sync" -void Resource::Sync( http::Agent *http, DateTime& sync_time, const Val& options ) +void Resource::Sync( Syncer *syncer, DateTime& sync_time, const Val& options ) { assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced - SyncSelf( http, options ) ; + SyncSelf( syncer, options ) ; // we want the server sync time, so we will take the server time of the last file uploaded to store as the sync time // m_mtime is updated to server modified time when the file is uploaded @@ -360,13 +368,13 @@ void Resource::Sync( http::Agent *http, DateTime& sync_time, const Val& options // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) std::for_each( m_child.begin(), m_child.end(), - boost::bind( &Resource::Sync, _1, http, boost::ref(sync_time), options ) ) ; + boost::bind( &Resource::Sync, _1, syncer, boost::ref(sync_time), options ) ) ; } -void Resource::SyncSelf( http::Agent* http, const Val& options ) +void Resource::SyncSelf( Syncer* syncer, const Val& options ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync - assert( IsRoot() || http == 0 || m_parent->IsFolder() ) ; + assert( IsRoot() || !syncer || m_parent->IsFolder() ) ; assert( IsRoot() || m_parent->m_state != remote_deleted ) ; assert( IsRoot() || m_parent->m_state != local_deleted ) ; @@ -377,30 +385,30 @@ void Resource::SyncSelf( http::Agent* http, const Val& options ) case local_new : Log( "sync %1% doesn't exist in server, uploading", path, log::info ) ; - if ( http != 0 && Create( http ) ) + if ( syncer && syncer->Create( this ) ) m_state = sync ; break ; case local_deleted : Log( "sync %1% deleted in local. deleting remote", path, log::info ) ; - if ( http != 0 ) - DeleteRemote( http ) ; + if ( syncer ) + syncer->DeleteRemote( this ) ; break ; case local_changed : Log( "sync %1% changed in local. uploading", path, log::info ) ; - if ( http != 0 && EditContent( http, options["new-rev"].Bool() ) ) + if ( syncer && syncer->EditContent( this, options["new-rev"].Bool() ) ) m_state = sync ; break ; case remote_new : Log( "sync %1% created in remote. creating local", path, log::info ) ; - if ( http != 0 ) + if ( syncer ) { if ( IsFolder() ) fs::create_directories( path ) ; else - Download( http, path ) ; + syncer->Download( this, path ) ; m_state = sync ; } @@ -409,16 +417,16 @@ void Resource::SyncSelf( http::Agent* http, const Val& options ) case remote_changed : assert( !IsFolder() ) ; Log( "sync %1% changed in remote. downloading", path, log::info ) ; - if ( http != 0 ) + if ( syncer ) { - Download( http, path ) ; + syncer->Download( this, path ) ; m_state = sync ; } break ; case remote_deleted : Log( "sync %1% deleted in remote. deleting local", path, log::info ) ; - if ( http != 0 ) + if ( syncer ) DeleteLocal() ; break ; @@ -459,193 +467,6 @@ void Resource::DeleteLocal() } } -void Resource::DeleteRemote( http::Agent *http ) -{ - assert( http != 0 ) ; - http::StringResponse str ; - - try - { - http::Header hdr ; - hdr.Add( "If-Match: " + m_etag ) ; - - // doesn't know why, but an update before deleting seems to work always - http::XmlResponse xml ; - http->Get( m_href, &xml, hdr ) ; - AssignIDs( Entry1( xml.Response() ) ) ; - - http->Custom( "DELETE", m_href, &str, hdr ) ; - } - catch ( Exception& e ) - { - // don't rethrow here. there are some cases that I don't know why - // the delete will fail. - Trace( "Exception %1% %2%", - boost::diagnostic_information(e), - str.Response() ) ; - } -} - - -void Resource::Download( http::Agent* http, const fs::path& file ) const -{ - assert( http != 0 ) ; - - http::Download dl( file.string(), http::Download::NoChecksum() ) ; - long r = http->Get( m_content, &dl, http::Header() ) ; - if ( r <= 400 ) - { - if ( m_mtime != DateTime() ) - os::SetFileTime( file, m_mtime ) ; - else - Log( "encountered zero date time after downloading %1%", file, log::warning ) ; - } -} - -bool Resource::EditContent( http::Agent* http, bool new_rev ) -{ - assert( http != 0 ) ; - assert( m_parent != 0 ) ; - assert( m_parent->m_state == sync ) ; - - // upload link missing means that file is read only - if ( m_edit.empty() ) - { - Log( "Cannot upload %1%: file read-only. %2%", m_name, m_state, log::warning ) ; - return false ; - } - - return Upload( http, m_edit + (new_rev ? "?new-revision=true" : ""), false ) ; -} - -bool Resource::Create( http::Agent* http ) -{ - assert( http != 0 ) ; - assert( m_parent != 0 ) ; - assert( m_parent->IsFolder() ) ; - assert( m_parent->m_state == sync ) ; - - if ( IsFolder() ) - { - std::string uri = feed_base ; - if ( !m_parent->IsRoot() ) - uri += ("/" + http->Escape(m_parent->m_id) + "/contents") ; - - std::string meta = (boost::format( xml_meta ) - % "folder" - % xml::Escape(m_name) - ).str() ; - - http::Header hdr ; - hdr.Add( "Content-Type: application/atom+xml" ) ; - - http::XmlResponse xml ; -// http::ResponseLog log( "create", ".xml", &xml ) ; - http->Post( uri, meta, &xml, hdr ) ; - AssignIDs( Entry1( xml.Response() ) ) ; - - return true ; - } - else if ( !m_parent->m_create.empty() ) - { - return Upload( http, m_parent->m_create + "?convert=false", true ) ; - } - else - { - Log( "parent of %1% does not exist: cannot upload", Name(), log::warning ) ; - return false ; - } -} - -bool Resource::Upload( - http::Agent* http, - const std::string& link, - bool post) -{ - assert( http != 0 ) ; - - File file( Path() ) ; - std::ostringstream xcontent_len ; - xcontent_len << "X-Upload-Content-Length: " << file.Size() ; - - http::Header hdr ; - hdr.Add( "Content-Type: application/atom+xml" ) ; - hdr.Add( "X-Upload-Content-Type: application/octet-stream" ) ; - hdr.Add( xcontent_len.str() ) ; - hdr.Add( "If-Match: " + m_etag ) ; - hdr.Add( "Expect:" ) ; - - std::string meta = (boost::format( xml_meta ) - % m_kind - % xml::Escape(m_name) - ).str() ; - - bool retrying=false; - while ( true ) { - if ( retrying ) { - file.Seek( 0, SEEK_SET ); - os::Sleep( 5 ); - } - - try { - http::StringResponse str ; - if ( post ) - http->Post( link, meta, &str, hdr ) ; - else - http->Put( link, meta, &str, hdr ) ; - } catch ( Error &e ) { - std::string const *info = boost::get_error_info(e); - if ( info && (*info == "XML_Parse") ) { - Log( "Error parsing pre-upload response XML, retrying whole upload in 5s", - log::warning ); - retrying = true; - continue; - } else { - throw e; - } - } - - http::Header uphdr ; - uphdr.Add( "Expect:" ) ; - uphdr.Add( "Accept:" ) ; - - // the content upload URL is in the "Location" HTTP header - std::string uplink = http->RedirLocation() ; - http::XmlResponse xml ; - - long http_code = 0; - try { - http_code = http->Put( uplink, &file, &xml, uphdr ) ; - } catch ( Error &e ) { - std::string const *info = boost::get_error_info(e); - if ( info && (*info == "XML_Parse") ) { - Log( "Error parsing response XML, retrying whole upload in 5s", - log::warning ); - retrying = true; - continue; - } else { - throw e; - } - } - - if ( http_code == 410 || http_code == 412 ) { - Log( "request failed with %1%, retrying whole upload in 5s", http_code, - log::warning ) ; - retrying = true; - continue; - } - - if ( retrying ) - Log( "upload succeeded on retry", log::warning ); - Entry1 responseEntry = Entry1( xml.Response() ); - AssignIDs( responseEntry ) ; - m_mtime = responseEntry.MTime(); - break; - } - - return true ; -} - Resource::iterator Resource::begin() const { return m_child.begin() ; @@ -694,4 +515,4 @@ bool Resource::HasID() const return !m_href.empty() && !m_id.empty() ; } -} } // end of namespace +} // end of namespace diff --git a/libgrive/src/drive/Resource.hh b/libgrive/src/base/Resource.hh similarity index 86% rename from libgrive/src/drive/Resource.hh rename to libgrive/src/base/Resource.hh index 62cea693..efb0d5f1 100644 --- a/libgrive/src/drive/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -29,17 +29,12 @@ namespace gr { -namespace http -{ - class Agent ; -} +class Syncer ; class Val ; class Entry ; -namespace v1 { - /*! \brief A resource can be a file or a folder in the google drive The google drive contains a number of resources, which is represented by this class. @@ -48,20 +43,56 @@ namespace v1 { class Resource { public : - struct Error : virtual Exception {} ; - typedef std::vector Children ; typedef Children::const_iterator iterator ; + /// State of the resource. indicating what to do with the resource + enum State + { + /// The best state: the file is the same in remote and in local. + sync, + + /// Resource created in local, but remote does not have it. + /// We should create the resource in google drive and upload new content + local_new, + + /// Resource exists in both local & remote, but changes in local is newer + /// than remote. We should upload local copy to overwrite remote. + local_changed, + + /// Resource deleted from local since last time grive has checked. + local_deleted, + + /// Resource created in google drive, but not exist in local. + /// We should download the file. + remote_new, + + /// Resource exists in both local & remote, but remote is newer. + remote_changed, + + /// Resource delete in remote, need to delete in local + remote_deleted, + + + /// invalid value + unknown + } ; + public : Resource(const fs::path& root_folder) ; Resource( const std::string& name, const std::string& kind ) ; bool IsFolder() const ; + bool IsEditable() const ; std::string Name() const ; + std::string Kind() const ; + DateTime MTime() const ; std::string SelfHref() const ; + std::string ContentSrc() const ; + std::string ETag() const ; std::string ResourceID() const ; + State GetState() const; const Resource* Parent() const ; Resource* Parent() ; @@ -77,7 +108,7 @@ public : void FromRemote( const Entry& remote, const DateTime& last_sync ) ; void FromLocal( const DateTime& last_sync ) ; - void Sync( http::Agent* http, DateTime& sync_time, const Val& options ) ; + void Sync( Syncer* syncer, DateTime& sync_time, const Val& options ) ; // children access iterator begin() const ; @@ -85,77 +116,41 @@ public : std::size_t size() const ; std::string StateStr() const ; - + private : - /// State of the resource. indicating what to do with the resource - enum State - { - /// The best state: the file is the same in remote and in local. - sync, - - /// Resource created in local, but remote does not have it. - /// We should create the resource in google drive and upload new content - local_new, - - /// Resource exists in both local & remote, but changes in local is newer - /// than remote. We should upload local copy to overwrite remote. - local_changed, - - /// Resource deleted from local since last time grive has checked. - local_deleted, - - /// Resource created in google drive, but not exist in local. - /// We should download the file. - remote_new, - - /// Resource exists in both local & remote, but remote is newer. - remote_changed, - - /// Resource delete in remote, need to delete in local - remote_deleted, - - - /// invalid value - unknown - } ; + + void AssignIDs( const Entry& remote ) ; friend std::ostream& operator<<( std::ostream& os, State s ) ; - + friend class Syncer ; + private : void SetState( State new_state ) ; - - void Download( http::Agent* http, const fs::path& file ) const ; - bool EditContent( http::Agent* http, bool new_rev ) ; - bool Create( http::Agent* http ) ; - bool Upload( http::Agent* http, const std::string& link, bool post ) ; void FromRemoteFolder( const Entry& remote, const DateTime& last_sync ) ; void FromRemoteFile( const Entry& remote, const DateTime& last_sync ) ; void DeleteLocal() ; - void DeleteRemote( http::Agent* http ) ; - - void AssignIDs( const Entry& remote ) ; - void SyncSelf( http::Agent* http, const Val& options ) ; + void SyncSelf( Syncer* syncer, const Val& options ) ; + private : std::string m_name ; std::string m_kind ; std::string m_md5 ; DateTime m_mtime ; - + std::string m_id ; std::string m_href ; - std::string m_edit ; - std::string m_create ; std::string m_content ; std::string m_etag ; + bool m_is_editable ; // not owned Resource *m_parent ; std::vector m_child ; - + State m_state ; } ; -} } // end of namespace gr::v1 +} // end of namespace gr::v1 diff --git a/libgrive/src/base/Syncer.cc b/libgrive/src/base/Syncer.cc new file mode 100644 index 00000000..ef6d7de3 --- /dev/null +++ b/libgrive/src/base/Syncer.cc @@ -0,0 +1,42 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "Syncer.hh" +#include "Resource.hh" +#include "Entry.hh" +#include "http/Agent.hh" + +namespace gr { + +Syncer::Syncer( http::Agent *http ): + m_http( http ) +{ +} + +void Syncer::AssignIDs( Resource *res, const Entry& remote ) +{ + res->AssignIDs( remote ); +} + +void Syncer::AssignMTime( Resource *res, const DateTime& mtime ) +{ + res->m_mtime = mtime; +} + +} // end of namespace gr diff --git a/libgrive/src/base/Syncer.hh b/libgrive/src/base/Syncer.hh new file mode 100644 index 00000000..6665e5ca --- /dev/null +++ b/libgrive/src/base/Syncer.hh @@ -0,0 +1,62 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "util/FileSystem.hh" + +#include +#include +#include + +namespace gr { + +namespace http +{ + class Agent ; +} + +class DateTime ; + +class Resource ; + +class Entry ; + +/*! \brief A Syncer incapsulates all resource-related upload/download/edit methods */ +class Syncer +{ +public : + + Syncer( http::Agent *http ); + + virtual void DeleteRemote( Resource *res ) = 0; + virtual void Download( Resource *res, const fs::path& file ) = 0; + virtual bool EditContent( Resource *res, bool new_rev ) = 0; + virtual bool Create( Resource *res ) = 0; + +protected: + + http::Agent *m_http; + + void AssignIDs( Resource *res, const Entry& remote ); + void AssignMTime( Resource *res, const DateTime& mtime ); + +} ; + +} // end of namespace gr diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index a8b99b62..e31c2c83 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -22,6 +22,7 @@ #include "CommonUri.hh" #include "base/Entry.hh" #include "Feed.hh" +#include "Syncer1.hh" #include "http/Agent.hh" #include "http/ResponseLog.hh" @@ -107,7 +108,7 @@ void Drive::SyncFolders( ) // first, get all collections from the query result for ( Feed::iterator i = feed.begin() ; i != feed.end() ; ++i ) { - Entry e( *i ) ; + const Entry &e = *i ; if ( e.IsDir() ) { if ( e.ParentHrefs().size() != 1 ) @@ -171,8 +172,10 @@ void Drive::DetectChanges() void Drive::Update() { + Syncer1 syncer( m_http ); + Log( "Synchronizing files", log::info ) ; - m_state.Sync( m_http, m_options ) ; + m_state.Sync( &syncer, m_options ) ; UpdateChangeStamp( ) ; } diff --git a/libgrive/src/drive/ResourceTree.hh b/libgrive/src/drive/ResourceTree.hh index d0a391c9..1c341723 100644 --- a/libgrive/src/drive/ResourceTree.hh +++ b/libgrive/src/drive/ResourceTree.hh @@ -19,7 +19,7 @@ #pragma once -#include "Resource.hh" +#include "base/Resource.hh" #include "util/FileSystem.hh" diff --git a/libgrive/src/drive/State.cc b/libgrive/src/drive/State.cc index d172fcc2..a74d414d 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/drive/State.cc @@ -20,10 +20,10 @@ #include "State.hh" #include "base/Entry.hh" -#include "Resource.hh" +#include "base/Resource.hh" +#include "base/Syncer.hh" #include "CommonUri.hh" -#include "http/Agent.hh" #include "util/Crypt.hh" #include "util/File.hh" #include "util/log/Log.hh" @@ -273,7 +273,7 @@ void State::Write( const fs::path& filename ) const fs << result ; } -void State::Sync( http::Agent *http, const Val& options ) +void State::Sync( Syncer *syncer, const Val& options ) { // set the last sync time from the time returned by the server for the last file synced // if the sync time hasn't changed (i.e. now files have been uploaded) @@ -283,7 +283,7 @@ void State::Sync( http::Agent *http, const Val& options ) // TODO - WARNING - do we use the last sync time to compare to client file times // need to check if this introduces a new problem DateTime last_sync_time = m_last_sync; - m_res.Root()->Sync( http, last_sync_time, options ) ; + m_res.Root()->Sync( syncer, last_sync_time, options ) ; if ( last_sync_time == m_last_sync ) { diff --git a/libgrive/src/drive/State.hh b/libgrive/src/drive/State.hh index f2acfb98..3fe68ce3 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/drive/State.hh @@ -28,19 +28,16 @@ namespace gr { -namespace http -{ - class Agent ; -} - class Val ; class Entry ; -namespace v1 { +class Syncer ; class Resource ; +namespace v1 { + class State { public : @@ -60,7 +57,7 @@ public : Resource* FindByHref( const std::string& href ) ; Resource* FindByID( const std::string& id ) ; - void Sync( http::Agent *http, const Val& options ) ; + void Sync( Syncer *syncer, const Val& options ) ; iterator begin() ; iterator end() ; diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc new file mode 100644 index 00000000..816572fa --- /dev/null +++ b/libgrive/src/drive/Syncer1.cc @@ -0,0 +1,255 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "base/Resource.hh" +#include "CommonUri.hh" +#include "Entry1.hh" +#include "Syncer1.hh" + +#include "http/Agent.hh" +#include "http/Download.hh" +#include "http/Header.hh" +//#include "http/ResponseLog.hh" +#include "http/StringResponse.hh" +#include "http/XmlResponse.hh" + +#include "xml/Node.hh" +#include "xml/NodeSet.hh" +#include "xml/String.hh" +#include "xml/TreeBuilder.hh" + +#include "util/OS.hh" +#include "util/log/Log.hh" + +#include + +#include + +// for debugging +#include + +namespace gr { namespace v1 { + +// hard coded XML file +const std::string xml_meta = + "\n" + "" + "" + "%2%" + "" ; + +Syncer1::Syncer1( http::Agent *http ): + Syncer( http ) +{ +} + +void Syncer1::DeleteRemote( Resource *res ) +{ + http::StringResponse str ; + + try + { + http::Header hdr ; + hdr.Add( "If-Match: " + res->ETag() ) ; + + // don't know why, but an update before deleting seems to work always + http::XmlResponse xml ; + m_http->Get( res->SelfHref(), &xml, hdr ) ; + AssignIDs( res, Entry1( xml.Response() ) ) ; + + m_http->Custom( "DELETE", res->SelfHref(), &str, hdr ) ; + } + catch ( Exception& e ) + { + // don't rethrow here. there are some cases that I don't know why + // the delete will fail. + Trace( "Exception %1% %2%", + boost::diagnostic_information(e), + str.Response() ) ; + } +} + +void Syncer1::Download( Resource *res, const fs::path& file ) +{ + http::Download dl( file.string(), http::Download::NoChecksum() ) ; + long r = m_http->Get( res->ContentSrc(), &dl, http::Header() ) ; + if ( r <= 400 ) + { + if ( res->MTime() != DateTime() ) + os::SetFileTime( file, res->MTime() ) ; + else + Log( "encountered zero date time after downloading %1%", file, log::warning ) ; + } +} + +bool Syncer1::EditContent( Resource *res, bool new_rev ) +{ + assert( res->Parent() ) ; + assert( res->Parent()->GetState() == Resource::sync ) ; + + if ( !res->IsEditable() ) + { + Log( "Cannot upload %1%: file read-only. %2%", res->Name(), res->StateStr(), log::warning ) ; + return false ; + } + + return Upload( res, feed_base + "/" + res->ResourceID() + ( new_rev ? "?new-revision=true" : "" ), false ) ; +} + +bool Syncer1::Create( Resource *res ) +{ + assert( res->Parent() ) ; + assert( res->Parent()->IsFolder() ) ; + assert( res->Parent()->GetState() == Resource::sync ) ; + + if ( res->IsFolder() ) + { + std::string uri = feed_base ; + if ( !res->Parent()->IsRoot() ) + uri += ( "/" + m_http->Escape( res->Parent()->ResourceID() ) + "/contents" ) ; + + std::string meta = (boost::format( xml_meta ) + % "folder" + % xml::Escape( res->Name() ) + ).str() ; + + http::Header hdr ; + hdr.Add( "Content-Type: application/atom+xml" ) ; + + http::XmlResponse xml ; +// http::ResponseLog log( "create", ".xml", &xml ) ; + m_http->Post( uri, meta, &xml, hdr ) ; + AssignIDs( res, Entry1( xml.Response() ) ) ; + + return true ; + } + else if ( res->Parent()->IsEditable() ) + { + return Upload( res, root_create + (res->Parent()->ResourceID() == "folder:root" + ? "" : "/" + res->Parent()->ResourceID() + "/contents") + "?convert=false", true ) ; + } + else + { + Log( "parent of %1% does not exist: cannot upload", res->Name(), log::warning ) ; + return false ; + } +} + +bool Syncer1::Upload( Resource *res, + const std::string& link, + bool post ) +{ + File file( res->Path() ) ; + std::ostringstream xcontent_len ; + xcontent_len << "X-Upload-Content-Length: " << file.Size() ; + + http::Header hdr ; + hdr.Add( "Content-Type: application/atom+xml" ) ; + hdr.Add( "X-Upload-Content-Type: application/octet-stream" ) ; + hdr.Add( xcontent_len.str() ) ; + hdr.Add( "If-Match: " + res->ETag() ) ; + hdr.Add( "Expect:" ) ; + + std::string meta = (boost::format( xml_meta ) + % res->Kind() + % xml::Escape( res->Name() ) + ).str() ; + + bool retrying = false; + while ( true ) + { + if ( retrying ) + { + file.Seek( 0, SEEK_SET ); + os::Sleep( 2 ); + } + + try + { + http::StringResponse str ; + if ( post ) + m_http->Post( link, meta, &str, hdr ) ; + else + m_http->Put( link, meta, &str, hdr ) ; + } + catch ( Exception &e ) + { + std::string const *info = boost::get_error_info(e); + if ( info && (*info == "XML_Parse") ) + { + Log( "Error parsing pre-upload response XML, retrying whole upload in 5s", + log::warning ); + retrying = true; + continue; + } + else + { + throw e; + } + } + + http::Header uphdr ; + uphdr.Add( "Expect:" ) ; + uphdr.Add( "Accept:" ) ; + + // the content upload URL is in the "Location" HTTP header + std::string uplink = m_http->RedirLocation() ; + http::XmlResponse xml ; + + long http_code = 0; + try + { + http_code = m_http->Put( uplink, &file, &xml, uphdr ) ; + } + catch ( Exception &e ) + { + std::string const *info = boost::get_error_info(e); + if ( info && (*info == "XML_Parse") ) + { + Log( "Error parsing response XML, retrying whole upload in 5s", + log::warning ); + retrying = true; + continue; + } + else + { + throw e; + } + } + + if ( http_code == 410 || http_code == 412 ) + { + Log( "request failed with %1%, retrying whole upload in 5s", http_code, log::warning ) ; + retrying = true; + continue; + } + + if ( retrying ) + Log( "upload succeeded on retry", log::warning ); + Entry1 responseEntry = Entry1( xml.Response() ); + AssignIDs( res, responseEntry ) ; + AssignMTime( res, responseEntry.MTime() ); + break; + } + + return true ; +} + +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Syncer1.hh b/libgrive/src/drive/Syncer1.hh new file mode 100644 index 00000000..7f3d1cc4 --- /dev/null +++ b/libgrive/src/drive/Syncer1.hh @@ -0,0 +1,46 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "base/Syncer.hh" + +namespace gr { + +namespace v1 { + +class Syncer1: public Syncer +{ + +public : + + Syncer1( http::Agent *http ); + + void DeleteRemote( Resource *res ); + void Download( Resource *res, const fs::path& file ); + bool EditContent( Resource *res, bool new_rev ); + bool Create( Resource *res ); + +private : + + bool Upload( Resource *res, const std::string& link, bool post); + +} ; + +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive2/Drive.cc b/libgrive/src/drive2/Drive.cc deleted file mode 100644 index feaa84be..00000000 --- a/libgrive/src/drive2/Drive.cc +++ /dev/null @@ -1,164 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#include "Drive.hh" - -#include "CommonUri.hh" -#include "Feed.hh" -#include "json/Val.hh" -#include "util/Exception.hh" - -#include - -#include -#include -#include - -#include - -namespace gr { namespace v2 { - -Drive::Drive( ) : - m_root( 0 ) -{ - -} - -void Drive::Refresh( http::Agent *agent ) -{ - // find root node ID - assert( m_root == 0 ) ; - m_root = NewResource( agent, "root" ) ; - - // get all folders first - Feed folders( feeds::files ) ; - folders.Query( "mimeType", mime_types::folder ) ; - NewResource( agent, folders ) ; - - // get all files - Feed files( feeds::files ) ; - NewResource( agent, files ) ; - - // build parent-child linkage between folders - for ( details::DB::iterator i = m_db.begin() ; i != m_db.end() ; ++i ) - { - Resource *parent = Find( (*i)->Parent() ), *child = *i ; - assert( child != 0 ) ; - - if ( parent != 0 ) - { - // initialize parent IDs - parent->AddChild( child->ID() ) ; - child->SetParent( parent->ID() ) ; - } - } -} - -void Drive::NewResource( http::Agent *agent, Feed& items ) -{ - assert( agent != 0 ) ; - - while ( items.Next( agent ) ) - { - std::vector item_json = items.Content()["items"].AsArray() ; - for ( std::vector::iterator i = item_json.begin() ; i != item_json.end() ; ++i ) - NewResource( *i ) ; - } -} - -/// Create resource base on ID -Resource* Drive::NewResource( http::Agent *agent, const std::string& id ) -{ - Feed feed( feeds::files + "/" + id ) ; - feed.Next( agent ) ; - - return NewResource( feed.Content() ) ; -} - -Resource* Drive::NewResource( const Val& item ) -{ - // assume resource is directly under root - std::string parent_id = m_root != 0 ? m_root->ID() : "" ; - - Val parents ; - if ( item.Get( "parents", parents ) ) - { - std::vector pids_val = parents.Select( "id" ) ; - std::vector pids ; - std::transform( pids_val.begin(), pids_val.end(), - std::back_inserter( pids ), - boost::bind( &Val::Str, _1 ) ) ; - - // only the first parent counts - if ( !pids.empty() ) - parent_id = pids.front() ; - } - - Resource *r = new Resource( - item["id"].Str(), - item["mimeType"].Str(), - item["title"].Str(), - parent_id ) ; - - m_db.insert(r) ; - assert( Find(r->ID()) == r ) ; - - return r ; -} - -Resource* Drive::Find( const std::string& id ) -{ - details::ID::iterator i = m_db.get().find(id) ; - return i != m_db.get().end() ? *i : 0 ; -} - -const Resource* Drive::Find( const std::string& id ) const -{ - details::ID::const_iterator i = m_db.get().find(id) ; - return i != m_db.get().end() ? *i : 0 ; -} - -Resource* Drive::Root() -{ - return m_root ; -} - -const Resource* Drive::Root() const -{ - return m_root ; -} - -const Resource* Drive::Child( const Resource *parent, std::size_t idx ) const -{ - if ( idx >= parent->ChildCount() ) - BOOST_THROW_EXCEPTION( - Exception() - ) ; - - return Find( parent->At(idx) ) ; -} - -const Resource* Drive::Parent( const Resource *child ) const -{ - return Find( child->Parent() ) ; -} - -} } // end of namespace gr::v2 - diff --git a/libgrive/src/drive2/Drive.hh b/libgrive/src/drive2/Drive.hh deleted file mode 100644 index 33014b03..00000000 --- a/libgrive/src/drive2/Drive.hh +++ /dev/null @@ -1,89 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#pragma once - -#include "Resource.hh" - -#include -#include -#include -#include - -namespace gr { - -namespace http -{ - class Agent ; -} - -class Val ; - -namespace v2 { - -namespace details -{ - using namespace boost::multi_index ; - struct ByID {} ; - struct ByHref {} ; - struct ByIdentity {} ; - - typedef multi_index_container< - Resource*, - indexed_by< - hashed_unique, identity >, - hashed_non_unique, const_mem_fun > - > - > DB ; - - typedef DB::index::type ID ; - typedef DB::index::type Set ; -} - -class Feed ; - -class Drive -{ -public : - Drive( ) ; - - void Refresh( http::Agent *agent ) ; - - Resource* Find( const std::string& id ) ; - const Resource* Find( const std::string& id ) const ; - - Resource* Root() ; - const Resource* Root() const ; - - const Resource* Child( const Resource *parent, std::size_t idx ) const ; - const Resource* Parent( const Resource *child ) const ; - -private : - Resource* NewResource( const Val& item ) ; - Resource* NewResource( http::Agent *agent, const std::string& id ) ; - void NewResource( http::Agent *agent, Feed& items ) ; - -private : - details::DB m_db ; - - Resource *m_root ; -} ; - -} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Feed.cc b/libgrive/src/drive2/Feed.cc index 78f67f06..93f5a24b 100644 --- a/libgrive/src/drive2/Feed.cc +++ b/libgrive/src/drive2/Feed.cc @@ -34,6 +34,7 @@ Feed::Feed( const std::string& base ) : m_content.Add( "nextLink", Val(base) ) ; } +// for example to find dirs: Query( "mimeType", mime_types::folder ) void Feed::Query( const std::string& field, const std::string& value ) { std::string url = m_content["nextLink"].Str() ; diff --git a/libgrive/src/drive2/Resource.cc b/libgrive/src/drive2/Resource.cc deleted file mode 100644 index a5832792..00000000 --- a/libgrive/src/drive2/Resource.cc +++ /dev/null @@ -1,100 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#include "Resource.hh" - -#include "CommonUri.hh" - -#include - -namespace gr { namespace v2 { - -/** Default constructor construct the resource of the root folder -*/ -Resource::Resource() : - m_id( "root" ), - m_mime( mime_types::folder ), - m_title( "Root folder" ) -{ -} - -Resource::Resource( - const std::string& id, - const std::string& mime, - const std::string& title, - const std::string& parent ) : - m_id ( id ), - m_mime ( mime ), - m_title ( title ), - m_parent( parent ) -{ -} - -std::string Resource::ID() const -{ - return m_id ; -} - -std::string Resource::Mime() const -{ - return m_mime ; -} - -std::string Resource::Title() const -{ - return m_title ; -} - -bool Resource::IsFolder() const -{ - return m_mime == "application/vnd.google-apps.folder" ; -} - -void Resource::AddChild( const std::string& child ) -{ - m_children.push_back( child ) ; -} - -void Resource::SetParent( const std::string& parent ) -{ - m_parent = parent ; -} - -std::size_t Resource::ChildCount() const -{ - return m_children.size() ; -} - -std::string Resource::At( std::size_t idx ) const -{ - return m_children.at(idx) ; -} - -std::string Resource::Parent() const -{ - return m_parent ; -} - -std::size_t Resource::Index( const std::string& child ) const -{ - return std::find( m_children.begin(), m_children.end(), child ) - m_children.begin() ; -} - -} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Resource.hh b/libgrive/src/drive2/Resource.hh deleted file mode 100644 index 8513c604..00000000 --- a/libgrive/src/drive2/Resource.hh +++ /dev/null @@ -1,62 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2013 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - MA 02110-1301, USA. -*/ - -#pragma once - -#include -#include - -namespace gr { namespace v2 { - -class Resource -{ -public : - Resource() ; - Resource( - const std::string& id, - const std::string& mime, - const std::string& title, - const std::string& parent ) ; - - std::string ID() const ; - std::string Mime() const ; - std::string Title() const ; - - bool IsFolder() const ; - - void AddChild( const std::string& child ) ; - void SetParent( const std::string& parent ) ; - - std::size_t ChildCount() const ; - std::string At( std::size_t idx ) const ; - std::string Parent() const ; - std::size_t Index( const std::string& child ) const ; - -private : - std::string m_id ; - std::string m_mime ; - std::string m_title ; - - std::vector m_children ; - - std::string m_parent ; -} ; - -} } // end of namespace gr::v2 From dabaaac38fca32d521fe9ea6cdf4ce5e7d32ff66 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 12:29:00 +0300 Subject: [PATCH 075/166] Cleanup Feed interface (remove some unused methods) --- libgrive/src/drive/Drive.cc | 9 ++------ libgrive/src/drive/Drive.hh | 2 -- libgrive/src/drive/Feed.cc | 23 ------------------- libgrive/src/drive/Feed.hh | 10 +-------- libgrive/src/drive2/Feed.cc | 44 ++++++++++++++----------------------- libgrive/src/drive2/Feed.hh | 18 ++++----------- 6 files changed, 23 insertions(+), 83 deletions(-) diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/drive/Drive.cc index e31c2c83..ea466549 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/drive/Drive.cc @@ -99,10 +99,8 @@ void Drive::SyncFolders( ) Log( "Synchronizing folders", log::info ) ; - http::XmlResponse xml ; - m_http->Get( feed_base + "/-/folder?max-results=50&showroot=true", &xml, http::Header() ) ; - - Feed feed( xml.Response() ) ; + Feed feed ; + feed.Start( m_http, feed_base + "/-/folder?max-results=50&showroot=true" ) ; do { // first, get all collections from the query result @@ -143,9 +141,6 @@ void Drive::DetectChanges() feed.Start( m_http, feed_base + "?showfolders=true&showroot=true" ) ; - m_resume_link = feed.Root()["link"]. - Find( "@rel", "http://schemas.google.com/g/2005#resumable-create-media" )["@href"] ; - do { std::for_each( diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/drive/Drive.hh index 57c24f9b..c11a744a 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/drive/Drive.hh @@ -53,14 +53,12 @@ public : private : void SyncFolders( ) ; - void file(); void FromRemote( const Entry& entry ) ; void FromChange( const Entry& entry ) ; void UpdateChangeStamp( ) ; private : http::Agent *m_http ; - std::string m_resume_link ; fs::path m_root ; State m_state ; Val m_options ; diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed.cc index f57545d0..1ce3563e 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed.cc @@ -37,18 +37,6 @@ Feed::Feed( ) { } -Feed::Feed( const xml::Node& root ) : - m_root ( root ), - m_entries ( m_root["entry"] ) -{ -} - -void Feed::Assign( const xml::Node& root ) -{ - m_root = root ; - m_entries = m_root["entry"] ; -} - Feed::iterator Feed::begin() const { return iterator( m_entries.begin() ) ; @@ -59,12 +47,6 @@ Feed::iterator Feed::end() const return iterator( m_entries.end() ) ; } -std::string Feed::Next() const -{ - xml::NodeSet nss = m_root["link"].Find( "@rel", "next" ) ; - return nss.empty() ? "" : std::string(nss["@href"]) ; -} - void Feed::Start( http::Agent *http, const std::string& url ) { http::XmlResponse xrsp ; @@ -120,9 +102,4 @@ void Feed::EnableLog( const std::string& prefix, const std::string& suffix ) m_log->sequence = 0 ; } -const xml::Node& Feed::Root() const -{ - return m_root ; -} - } } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/drive/Feed.hh index ce85435d..ef87a27a 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/drive/Feed.hh @@ -33,7 +33,6 @@ namespace gr { namespace http { class Agent ; - class Header ; } namespace v1 { @@ -44,21 +43,14 @@ public : class iterator ; public : - explicit Feed( const xml::Node& root ) ; Feed( ) ; void Start( http::Agent *http, const std::string& url ) ; - - void Assign( const xml::Node& root ) ; - const xml::Node& Root() const ; + bool GetNext( http::Agent *http ) ; iterator begin() const ; iterator end() const ; - std::string Next() const ; - bool GetNext( http::Agent *http ) ; - void EnableLog( const std::string& prefix, const std::string& suffix ) ; - private : struct LogInfo { diff --git a/libgrive/src/drive2/Feed.cc b/libgrive/src/drive2/Feed.cc index 93f5a24b..03fcea49 100644 --- a/libgrive/src/drive2/Feed.cc +++ b/libgrive/src/drive2/Feed.cc @@ -27,44 +27,32 @@ namespace gr { namespace v2 { -Feed::Feed( const std::string& base ) : - m_base( base ) +Feed::Feed( ) { - // Next() will grab this link - m_content.Add( "nextLink", Val(base) ) ; } -// for example to find dirs: Query( "mimeType", mime_types::folder ) -void Feed::Query( const std::string& field, const std::string& value ) +// for example to find dirs: '?q=mimeType%3d%27' + mime_types::folder + '%27' +void Feed::Start( http::Agent *http, const std::string& url ) { - std::string url = m_content["nextLink"].Str() ; - m_content.Add( "nextLink", Val( url + "?q=" + field + "+%3d+%27" + value + "%27" ) ) ; + http::ValResponse out ; + + http->Get( url, &out, http::Header() ) ; + + m_content = out.Response() ; } -bool Feed::Next( http::Agent *agent ) +bool Feed::GetNext( http::Agent *http ) { + assert( http != 0 ) ; + Val url ; - if ( !m_content.Get("nextLink", url) ) - return false ; - - http::ValResponse out ; - try + if ( m_content.Get( "nextLink", url ) ) { - agent->Get( url.Str(), &out, http::Header() ) ; + Start( http, url ) ; + return true ; } - catch ( Exception& e ) - { - e << DriveFeed_( m_content ) ; - throw ; - } - m_content = out.Response() ; - - return true ; -} - -Val Feed::Content() const -{ - return m_content ; + else + return false ; } } } // end of namespace diff --git a/libgrive/src/drive2/Feed.hh b/libgrive/src/drive2/Feed.hh index ad1f294a..71917509 100644 --- a/libgrive/src/drive2/Feed.hh +++ b/libgrive/src/drive2/Feed.hh @@ -29,7 +29,6 @@ namespace gr namespace http { class Agent ; - class Header ; } class Val ; @@ -39,21 +38,12 @@ namespace v2 { class Feed { public : - // exception info - typedef boost::error_info DriveFeed_ ; - -public : - Feed( const std::string& base ) ; - void Query( const std::string& field, const std::string& value ) ; - - - bool Next( http::Agent *agent ) ; - - Val Content() const ; + Feed( ) ; + void Start( http::Agent *http, const std::string& url ) ; + bool GetNext( http::Agent *http ) ; private : - std::string m_base ; - Val m_content ; + Val m_content ; } ; } } // end of namespace gr::v2 From 717a6a47930aeb431d728ebe0ecd4b47181e3092 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 14:32:04 +0300 Subject: [PATCH 076/166] Use vector instead of iterator_adaptor to hide implementation details --- libgrive/src/drive/Feed.cc | 35 ++++++++++++----------------------- libgrive/src/drive/Feed.hh | 28 +++++----------------------- 2 files changed, 17 insertions(+), 46 deletions(-) diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed.cc index 1ce3563e..087defde 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed.cc @@ -39,12 +39,12 @@ Feed::Feed( ) Feed::iterator Feed::begin() const { - return iterator( m_entries.begin() ) ; + return m_entries.begin() ; } Feed::iterator Feed::end() const { - return iterator( m_entries.end() ) ; + return m_entries.end() ; } void Feed::Start( http::Agent *http, const std::string& url ) @@ -60,8 +60,13 @@ void Feed::Start( http::Agent *http, const std::string& url ) http->Get( url, &log, http::Header() ) ; - m_root = xrsp.Response() ; - m_entries = m_root["entry"] ; + m_root = xrsp.Response() ; + xml::NodeSet xe = m_root["entry"] ; + m_entries.clear() ; + for ( xml::NodeSet::iterator i = xe.begin() ; i != xe.end() ; ++i ) + { + m_entries.push_back( Entry1( *i ) ); + } } bool Feed::GetNext( http::Agent *http ) @@ -78,28 +83,12 @@ bool Feed::GetNext( http::Agent *http ) return false ; } -Feed::iterator::iterator( ) -{ -} - -Feed::iterator::iterator( xml::Node::iterator i ) -{ - // for some reason, gcc 4.4.4 doesn't allow me to initialize the base class - // in the initializer. I have no choice but to initialize here. - base_reference() = i ; -} - -Feed::iterator::reference Feed::iterator::dereference() const -{ - return Entry1( *base_reference() ) ; -} - void Feed::EnableLog( const std::string& prefix, const std::string& suffix ) { m_log.reset( new LogInfo ) ; - m_log->prefix = prefix ; - m_log->suffix = suffix ; - m_log->sequence = 0 ; + m_log->prefix = prefix ; + m_log->suffix = suffix ; + m_log->sequence = 0 ; } } } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/drive/Feed.hh index ef87a27a..db27137a 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/drive/Feed.hh @@ -24,7 +24,7 @@ #include "xml/Node.hh" #include "xml/NodeSet.hh" -#include +#include #include @@ -40,13 +40,13 @@ namespace v1 { class Feed { public : - class iterator ; + typedef std::vector Entries; + typedef std::vector::const_iterator iterator; public : Feed( ) ; void Start( http::Agent *http, const std::string& url ) ; bool GetNext( http::Agent *http ) ; - iterator begin() const ; iterator end() const ; @@ -61,25 +61,7 @@ private : std::auto_ptr m_log ; xml::Node m_root ; - xml::NodeSet m_entries ; -} ; - -class Feed::iterator : public boost::iterator_adaptor< - Feed::iterator, - xml::Node::iterator, - Entry, - boost::random_access_traversal_tag, - Entry -> -{ -public : - iterator() ; - explicit iterator( xml::Node::iterator i ) ; - -private : - friend class boost::iterator_core_access; - - reference dereference() const ; + Entries m_entries ; } ; -} } // end of namespace +} } // end of namespace gr::v1 From 862fca035af0b1f953bfd57abc21c8479d91b8b3 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 16:54:04 +0300 Subject: [PATCH 077/166] Move all protocol interaction into Syncers out of the base code --- grive/src/main.cc | 6 +- libgrive/src/{drive => base}/Drive.cc | 77 ++++++--------- libgrive/src/{drive => base}/Drive.hh | 16 ++- libgrive/src/base/Feed.cc | 51 ++++++++++ libgrive/src/{drive => base}/Feed.hh | 31 +++--- libgrive/src/base/Resource.cc | 5 +- libgrive/src/{drive => base}/ResourceTree.cc | 9 +- libgrive/src/{drive => base}/ResourceTree.hh | 6 +- libgrive/src/{drive => base}/State.cc | 15 ++- libgrive/src/{drive => base}/State.hh | 4 +- libgrive/src/base/Syncer.cc | 5 + libgrive/src/base/Syncer.hh | 9 ++ libgrive/src/drive/CommonUri.hh | 2 - libgrive/src/drive/Entry1.cc | 7 +- libgrive/src/drive/{Feed.cc => Feed1.cc} | 50 +++------- libgrive/src/drive/{CommonUri.cc => Feed1.hh} | 23 +++-- libgrive/src/drive/Syncer1.cc | 31 ++++++ libgrive/src/drive/Syncer1.hh | 7 ++ libgrive/src/drive2/CommonUri.hh | 6 +- libgrive/src/drive2/Entry2.cc | 4 +- libgrive/src/drive2/{Feed.cc => Feed2.cc} | 40 ++++---- libgrive/src/drive2/{Feed.hh => Feed2.hh} | 22 +---- libgrive/src/drive2/Syncer2.cc | 97 +++++++++++++++++++ libgrive/src/drive2/Syncer2.hh | 51 ++++++++++ 24 files changed, 384 insertions(+), 190 deletions(-) rename libgrive/src/{drive => base}/Drive.cc (70%) rename libgrive/src/{drive => base}/Drive.hh (86%) create mode 100644 libgrive/src/base/Feed.cc rename libgrive/src/{drive => base}/Feed.hh (71%) rename libgrive/src/{drive => base}/ResourceTree.cc (95%) rename libgrive/src/{drive => base}/ResourceTree.hh (97%) rename libgrive/src/{drive => base}/State.cc (96%) rename libgrive/src/{drive => base}/State.hh (97%) rename libgrive/src/drive/{Feed.cc => Feed1.cc} (69%) rename libgrive/src/drive/{CommonUri.cc => Feed1.hh} (74%) rename libgrive/src/drive2/{Feed.cc => Feed2.cc} (60%) rename libgrive/src/drive2/{Feed.hh => Feed2.hh} (81%) create mode 100644 libgrive/src/drive2/Syncer2.cc create mode 100644 libgrive/src/drive2/Syncer2.hh diff --git a/grive/src/main.cc b/grive/src/main.cc index b172f409..b323c056 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -19,7 +19,8 @@ #include "util/Config.hh" -#include "drive/Drive.hh" +#include "base/Drive.hh" +#include "drive/Syncer1.hh" #include "http/CurlAgent.hh" #include "protocol/AuthAgent.hh" @@ -185,8 +186,9 @@ int Main( int argc, char **argv ) OAuth2 token( refresh_token, client_id, client_secret ) ; AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; + Syncer1 syncer( &agent ); - Drive drive( &agent, config.GetAll() ) ; + Drive drive( &syncer, config.GetAll() ) ; drive.DetectChanges() ; if ( vm.count( "dry-run" ) == 0 ) diff --git a/libgrive/src/drive/Drive.cc b/libgrive/src/base/Drive.cc similarity index 70% rename from libgrive/src/drive/Drive.cc rename to libgrive/src/base/Drive.cc index ea466549..10a194b4 100644 --- a/libgrive/src/drive/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -19,18 +19,13 @@ #include "Drive.hh" -#include "CommonUri.hh" -#include "base/Entry.hh" +#include "Entry.hh" #include "Feed.hh" -#include "Syncer1.hh" +#include "Syncer.hh" #include "http/Agent.hh" -#include "http/ResponseLog.hh" -#include "http/XmlResponse.hh" #include "util/Destroy.hh" #include "util/log/Log.hh" -#include "xml/Node.hh" -#include "xml/NodeSet.hh" #include @@ -45,20 +40,20 @@ // for debugging only #include -namespace gr { namespace v1 { +namespace gr { namespace { const std::string state_file = ".grive_state" ; } -Drive::Drive( http::Agent *agent, const Val& options ) : - m_http ( agent ), +Drive::Drive( Syncer *syncer, const Val& options ) : + m_syncer ( syncer ), m_root ( options["path"].Str() ), m_state ( m_root / state_file, options ), m_options ( options ) { - assert( m_http != 0 ) ; + assert( m_syncer ) ; } void Drive::FromRemote( const Entry& entry ) @@ -95,16 +90,13 @@ void Drive::SaveState() void Drive::SyncFolders( ) { - assert( m_http != 0 ) ; - Log( "Synchronizing folders", log::info ) ; - Feed feed ; - feed.Start( m_http, feed_base + "/-/folder?max-results=50&showroot=true" ) ; - do + std::auto_ptr feed = m_syncer->GetFolders() ; + while ( feed->GetNext( m_syncer->Agent() ) ) { // first, get all collections from the query result - for ( Feed::iterator i = feed.begin() ; i != feed.end() ; ++i ) + for ( Feed::iterator i = feed->begin() ; i != feed->end() ; ++i ) { const Entry &e = *i ; if ( e.IsDir() ) @@ -119,7 +111,7 @@ void Drive::SyncFolders( ) m_state.FromRemote( e ) ; } } - } while ( feed.GetNext( m_http ) ) ; + } m_state.ResolveEntry() ; } @@ -135,42 +127,38 @@ void Drive::DetectChanges() SyncFolders( ) ; Log( "Reading remote server file list", log::info ) ; - Feed feed ; + std::auto_ptr feed = m_syncer->GetAll() ; + if ( m_options["log-xml"].Bool() ) - feed.EnableLog( "/tmp/file", ".xml" ) ; + feed->EnableLog( "/tmp/file", ".xml" ) ; - feed.Start( m_http, feed_base + "?showfolders=true&showroot=true" ) ; - - do + while ( feed->GetNext( m_syncer->Agent() ) ) { std::for_each( - feed.begin(), feed.end(), + feed->begin(), feed->end(), boost::bind( &Drive::FromRemote, this, _1 ) ) ; - - } while ( feed.GetNext( m_http ) ) ; + } // pull the changes feed if ( prev_stamp != -1 ) { Log( "Detecting changes from last sync", log::info ) ; - Feed changes ; + feed = m_syncer->GetChanges( prev_stamp+1 ) ; if ( m_options["log-xml"].Bool() ) - feed.EnableLog( "/tmp/changes", ".xml" ) ; - - feed.Start( m_http, ChangesFeed(prev_stamp+1) ) ; - - std::for_each( - changes.begin(), changes.end(), - boost::bind( &Drive::FromChange, this, _1 ) ) ; + feed->EnableLog( "/tmp/changes", ".xml" ) ; + while ( feed->GetNext( m_syncer->Agent() ) ) + { + std::for_each( + feed->begin(), feed->end(), + boost::bind( &Drive::FromChange, this, _1 ) ) ; + } } } void Drive::Update() { - Syncer1 syncer( m_http ); - Log( "Synchronizing files", log::info ) ; - m_state.Sync( &syncer, m_options ) ; + m_state.Sync( m_syncer, m_options ) ; UpdateChangeStamp( ) ; } @@ -178,21 +166,14 @@ void Drive::Update() void Drive::DryRun() { Log( "Synchronizing files (dry-run)", log::info ) ; - m_state.Sync( 0, m_options ) ; + m_state.Sync( NULL, m_options ) ; } void Drive::UpdateChangeStamp( ) { - assert( m_http != 0 ) ; - - // get changed feed - http::XmlResponse xrsp ; - m_http->Get( ChangesFeed(m_state.ChangeStamp()+1), &xrsp, http::Header() ) ; - - // we should go through the changes to see if it was really Grive to made that change + // FIXME: we should go through the changes to see if it was really Grive to made that change // maybe by recording the updated timestamp and compare it? - m_state.ChangeStamp( - std::atoi(xrsp.Response()["docs:largestChangestamp"]["@value"].front().Value().c_str()) ) ; + m_state.ChangeStamp( m_syncer->GetChangeStamp( m_state.ChangeStamp()+1 ) ); } -} } // end of namespace gr::v1 +} // end of namespace gr diff --git a/libgrive/src/drive/Drive.hh b/libgrive/src/base/Drive.hh similarity index 86% rename from libgrive/src/drive/Drive.hh rename to libgrive/src/base/Drive.hh index c11a744a..7a63999f 100644 --- a/libgrive/src/drive/Drive.hh +++ b/libgrive/src/base/Drive.hh @@ -19,9 +19,8 @@ #pragma once -#include "State.hh" +#include "base/State.hh" -#include "http/Header.hh" #include "json/Val.hh" #include "util/Exception.hh" @@ -30,19 +29,16 @@ namespace gr { -namespace http -{ - class Agent ; -} +class Syncer ; class Entry ; -namespace v1 { +class State ; class Drive { public : - Drive( http::Agent *agent, const Val& options ) ; + Drive( Syncer *syncer, const Val& options ) ; void DetectChanges() ; void Update() ; @@ -58,10 +54,10 @@ private : void UpdateChangeStamp( ) ; private : - http::Agent *m_http ; + Syncer *m_syncer ; fs::path m_root ; State m_state ; Val m_options ; } ; -} } // end of namespace +} // end of namespace gr diff --git a/libgrive/src/base/Feed.cc b/libgrive/src/base/Feed.cc new file mode 100644 index 00000000..7ff38508 --- /dev/null +++ b/libgrive/src/base/Feed.cc @@ -0,0 +1,51 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "Feed.hh" + +#include "Entry.hh" + +#include "http/Agent.hh" + +namespace gr { + +Feed::Feed( const std::string &url ): + m_next( url ) +{ +} + +Feed::iterator Feed::begin() const +{ + return m_entries.begin() ; +} + +Feed::iterator Feed::end() const +{ + return m_entries.end() ; +} + +void Feed::EnableLog( const std::string& prefix, const std::string& suffix ) +{ + m_log.reset( new LogInfo ) ; + m_log->prefix = prefix ; + m_log->suffix = suffix ; + m_log->sequence = 0 ; +} + +} // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed.hh b/libgrive/src/base/Feed.hh similarity index 71% rename from libgrive/src/drive/Feed.hh rename to libgrive/src/base/Feed.hh index db27137a..8be1e55f 100644 --- a/libgrive/src/drive/Feed.hh +++ b/libgrive/src/base/Feed.hh @@ -21,9 +21,6 @@ #include "base/Entry.hh" -#include "xml/Node.hh" -#include "xml/NodeSet.hh" - #include #include @@ -35,8 +32,6 @@ namespace http class Agent ; } -namespace v1 { - class Feed { public : @@ -44,24 +39,22 @@ public : typedef std::vector::const_iterator iterator; public : - Feed( ) ; - void Start( http::Agent *http, const std::string& url ) ; - bool GetNext( http::Agent *http ) ; + Feed( const std::string& url ); + virtual bool GetNext( http::Agent *http ) = 0 ; iterator begin() const ; iterator end() const ; - - void EnableLog( const std::string& prefix, const std::string& suffix ) ; -private : + virtual void EnableLog( const std::string& prefix, const std::string& suffix ) ; + +protected : struct LogInfo { - std::string prefix ; - std::string suffix ; - std::size_t sequence ; + std::string prefix ; + std::string suffix ; + std::size_t sequence ; } ; - std::auto_ptr m_log ; - - xml::Node m_root ; - Entries m_entries ; + std::auto_ptr m_log ; + Entries m_entries ; + std::string m_next ; } ; -} } // end of namespace gr::v1 +} // end of namespace gr diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index ef995ab6..889e0706 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -38,7 +38,7 @@ namespace gr { /// default constructor creates the root folder -Resource::Resource(const fs::path& root_folder) : +Resource::Resource( const fs::path& root_folder ) : m_name ( root_folder.string() ), m_kind ( "folder" ), m_id ( "folder:root" ), @@ -339,7 +339,7 @@ fs::path Resource::Path() const bool Resource::IsInRootTree() const { assert( m_parent == 0 || m_parent->IsFolder() ) ; - return m_parent == 0 ? SelfHref().empty() : m_parent->IsInRootTree() ; + return m_parent == 0 ? IsRoot() : m_parent->IsInRootTree() ; } Resource* Resource::FindChild( const std::string& name ) @@ -507,6 +507,7 @@ std::string Resource::MD5() const bool Resource::IsRoot() const { + // Root entry does not show up in file feeds, so we check for empty parent (and self-href) return m_parent == 0 ; } diff --git a/libgrive/src/drive/ResourceTree.cc b/libgrive/src/base/ResourceTree.cc similarity index 95% rename from libgrive/src/drive/ResourceTree.cc rename to libgrive/src/base/ResourceTree.cc index fa8af7e3..3aa8d552 100644 --- a/libgrive/src/drive/ResourceTree.cc +++ b/libgrive/src/base/ResourceTree.cc @@ -18,7 +18,6 @@ */ #include "ResourceTree.hh" -#include "CommonUri.hh" #include "util/Destroy.hh" #include "util/log/Log.hh" @@ -26,12 +25,12 @@ #include #include -namespace gr { namespace v1 { +namespace gr { using namespace details ; ResourceTree::ResourceTree( const fs::path& rootFolder ) : - m_root(new Resource(rootFolder)) + m_root( new Resource( rootFolder ) ) { m_set.insert( m_root ) ; } @@ -43,7 +42,7 @@ ResourceTree::ResourceTree( const ResourceTree& fs ) : for ( Set::const_iterator i = s.begin() ; i != s.end() ; ++i ) { Resource *c = new Resource( **i ) ; - if ( c->SelfHref() == root_href ) + if ( c->IsRoot() ) m_root = c ; m_set.insert( c ) ; @@ -142,4 +141,4 @@ ResourceTree::iterator ResourceTree::end() return m_set.get().end() ; } -} } // end of namespace gr::v1 +} // end of namespace gr diff --git a/libgrive/src/drive/ResourceTree.hh b/libgrive/src/base/ResourceTree.hh similarity index 97% rename from libgrive/src/drive/ResourceTree.hh rename to libgrive/src/base/ResourceTree.hh index 1c341723..b74d9f19 100644 --- a/libgrive/src/drive/ResourceTree.hh +++ b/libgrive/src/base/ResourceTree.hh @@ -19,7 +19,7 @@ #pragma once -#include "base/Resource.hh" +#include "Resource.hh" #include "util/FileSystem.hh" @@ -30,8 +30,6 @@ namespace gr { -namespace v1 { - namespace details { using namespace boost::multi_index ; @@ -93,4 +91,4 @@ private : Resource* m_root ; } ; -} } // end of namespace gr::v1 +} // end of namespace gr diff --git a/libgrive/src/drive/State.cc b/libgrive/src/base/State.cc similarity index 96% rename from libgrive/src/drive/State.cc rename to libgrive/src/base/State.cc index a74d414d..d0525f2b 100644 --- a/libgrive/src/drive/State.cc +++ b/libgrive/src/base/State.cc @@ -19,10 +19,9 @@ #include "State.hh" -#include "base/Entry.hh" -#include "base/Resource.hh" -#include "base/Syncer.hh" -#include "CommonUri.hh" +#include "Entry.hh" +#include "Resource.hh" +#include "Syncer.hh" #include "util/Crypt.hh" #include "util/File.hh" @@ -32,11 +31,11 @@ #include -namespace gr { namespace v1 { +namespace gr { State::State( const fs::path& filename, const Val& options ) : - m_res ( options["path"].Str() ), - m_dir ( options["dir"].Str() ), + m_res ( options["path"].Str() ), + m_dir ( options["dir"].Str() ), m_cstamp ( -1 ) { Read( filename ) ; @@ -308,4 +307,4 @@ void State::ChangeStamp( long cstamp ) m_cstamp = cstamp ; } -} } // end of namespace gr::v1 +} // end of namespace gr diff --git a/libgrive/src/drive/State.hh b/libgrive/src/base/State.hh similarity index 97% rename from libgrive/src/drive/State.hh rename to libgrive/src/base/State.hh index 3fe68ce3..c870cae2 100644 --- a/libgrive/src/drive/State.hh +++ b/libgrive/src/base/State.hh @@ -36,8 +36,6 @@ class Syncer ; class Resource ; -namespace v1 { - class State { public : @@ -82,4 +80,4 @@ private : std::vector m_unresolved ; } ; -} } // end of namespace gr::v1 +} // end of namespace gr diff --git a/libgrive/src/base/Syncer.cc b/libgrive/src/base/Syncer.cc index ef6d7de3..98d2acad 100644 --- a/libgrive/src/base/Syncer.cc +++ b/libgrive/src/base/Syncer.cc @@ -29,6 +29,11 @@ Syncer::Syncer( http::Agent *http ): { } +http::Agent* Syncer::Agent() const +{ + return m_http; +} + void Syncer::AssignIDs( Resource *res, const Entry& remote ) { res->AssignIDs( remote ); diff --git a/libgrive/src/base/Syncer.hh b/libgrive/src/base/Syncer.hh index 6665e5ca..efe04bce 100644 --- a/libgrive/src/base/Syncer.hh +++ b/libgrive/src/base/Syncer.hh @@ -38,6 +38,8 @@ class Resource ; class Entry ; +class Feed ; + /*! \brief A Syncer incapsulates all resource-related upload/download/edit methods */ class Syncer { @@ -45,11 +47,18 @@ public : Syncer( http::Agent *http ); + http::Agent* Agent() const; + virtual void DeleteRemote( Resource *res ) = 0; virtual void Download( Resource *res, const fs::path& file ) = 0; virtual bool EditContent( Resource *res, bool new_rev ) = 0; virtual bool Create( Resource *res ) = 0; + virtual std::auto_ptr GetFolders() = 0; + virtual std::auto_ptr GetAll() = 0; + virtual std::auto_ptr GetChanges( long min_cstamp ) = 0; + virtual long GetChangeStamp( long min_cstamp ) = 0; + protected: http::Agent *m_http; diff --git a/libgrive/src/drive/CommonUri.hh b/libgrive/src/drive/CommonUri.hh index b0e06f7f..98cc9594 100644 --- a/libgrive/src/drive/CommonUri.hh +++ b/libgrive/src/drive/CommonUri.hh @@ -31,6 +31,4 @@ namespace gr { namespace v1 "https://docs.google.com/feeds/default/private/full/folder%3Aroot" ; const std::string root_create = "https://docs.google.com/feeds/upload/create-session/default/private/full" ; - - std::string ChangesFeed( int changestamp ) ; } } diff --git a/libgrive/src/drive/Entry1.cc b/libgrive/src/drive/Entry1.cc index 2d1d61a8..d44b421d 100644 --- a/libgrive/src/drive/Entry1.cc +++ b/libgrive/src/drive/Entry1.cc @@ -70,7 +70,12 @@ void Entry1::Update( const xml::Node& n ) m_parent_hrefs.clear( ) ; xml::NodeSet parents = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#parent" ) ; for ( xml::NodeSet::iterator i = parents.begin() ; i != parents.end() ; ++i ) - m_parent_hrefs.push_back( (*i)["@href"] ) ; + { + std::string href = (*i)["@href"]; + if ( href == root_href ) // API-independent root href is empty! + href = ""; + m_parent_hrefs.push_back( href ) ; + } // convert to lower case for easy comparison std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; diff --git a/libgrive/src/drive/Feed.cc b/libgrive/src/drive/Feed1.cc similarity index 69% rename from libgrive/src/drive/Feed.cc rename to libgrive/src/drive/Feed1.cc index 087defde..05093aaa 100644 --- a/libgrive/src/drive/Feed.cc +++ b/libgrive/src/drive/Feed1.cc @@ -17,7 +17,8 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "Feed.hh" +#include "CommonUri.hh" +#include "Feed1.hh" #include "Entry1.hh" @@ -33,62 +34,39 @@ namespace gr { namespace v1 { -Feed::Feed( ) +Feed1::Feed1( const std::string &url ): + Feed( url ) { } -Feed::iterator Feed::begin() const -{ - return m_entries.begin() ; -} - -Feed::iterator Feed::end() const -{ - return m_entries.end() ; -} - -void Feed::Start( http::Agent *http, const std::string& url ) +bool Feed1::GetNext( http::Agent *http ) { http::XmlResponse xrsp ; http::ResponseLog log( &xrsp ) ; + if ( m_next.empty() ) + return false; + if ( m_log.get() != 0 ) log.Reset( m_log->prefix, (boost::format( "-#%1%%2%" ) % m_log->sequence++ % m_log->suffix ).str(), &xrsp ) ; - http->Get( url, &log, http::Header() ) ; + http->Get( m_next, &log, http::Header() ) ; - m_root = xrsp.Response() ; + xml::Node m_root = xrsp.Response() ; xml::NodeSet xe = m_root["entry"] ; m_entries.clear() ; for ( xml::NodeSet::iterator i = xe.begin() ; i != xe.end() ; ++i ) { m_entries.push_back( Entry1( *i ) ); } -} - -bool Feed::GetNext( http::Agent *http ) -{ - assert( http != 0 ) ; - + xml::NodeSet nss = m_root["link"].Find( "@rel", "next" ) ; - if ( !nss.empty() ) - { - Start( http, nss["@href"] ) ; - return true ; - } - else - return false ; -} - -void Feed::EnableLog( const std::string& prefix, const std::string& suffix ) -{ - m_log.reset( new LogInfo ) ; - m_log->prefix = prefix ; - m_log->suffix = suffix ; - m_log->sequence = 0 ; + m_next = nss.empty() ? std::string( "" ) : nss["@href"]; + + return true; } } } // end of namespace gr::v1 diff --git a/libgrive/src/drive/CommonUri.cc b/libgrive/src/drive/Feed1.hh similarity index 74% rename from libgrive/src/drive/CommonUri.cc rename to libgrive/src/drive/Feed1.hh index a69b02b4..84466933 100644 --- a/libgrive/src/drive/CommonUri.cc +++ b/libgrive/src/drive/Feed1.hh @@ -17,15 +17,24 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "CommonUri.hh" -#include +#pragma once + +#include "base/Feed.hh" + +#include "xml/Node.hh" +#include "xml/NodeSet.hh" + +#include + +#include namespace gr { namespace v1 { -std::string ChangesFeed( int changestamp ) +class Feed1: public Feed { - boost::format feed( feed_changes + "?start-index=%1%" ) ; - return changestamp > 0 ? (feed%changestamp).str() : feed_changes ; -} +public : + Feed1( const std::string& url ) ; + bool GetNext( http::Agent *http ) ; +} ; -} } +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index 816572fa..3cde4172 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -20,6 +20,7 @@ #include "base/Resource.hh" #include "CommonUri.hh" #include "Entry1.hh" +#include "Feed1.hh" #include "Syncer1.hh" #include "http/Agent.hh" @@ -58,6 +59,7 @@ const std::string xml_meta = Syncer1::Syncer1( http::Agent *http ): Syncer( http ) { + assert( http != 0 ) ; } void Syncer1::DeleteRemote( Resource *res ) @@ -252,4 +254,33 @@ bool Syncer1::Upload( Resource *res, return true ; } +std::auto_ptr Syncer1::GetFolders() +{ + return std::auto_ptr( new Feed1( feed_base + "/-/folder?max-results=50&showroot=true" ) ); +} + +std::auto_ptr Syncer1::GetAll() +{ + return std::auto_ptr( new Feed1( feed_base + "?showfolders=true&showroot=true" ) ); +} + +std::string ChangesFeed( int changestamp ) +{ + boost::format feed( feed_changes + "?start-index=%1%" ) ; + return changestamp > 0 ? ( feed % changestamp ).str() : feed_changes ; +} + +std::auto_ptr Syncer1::GetChanges( long min_cstamp ) +{ + return std::auto_ptr( new Feed1( ChangesFeed( min_cstamp ) ) ); +} + +long Syncer1::GetChangeStamp( long min_cstamp ) +{ + http::XmlResponse xrsp ; + m_http->Get( ChangesFeed( min_cstamp ), &xrsp, http::Header() ) ; + + return std::atoi( xrsp.Response()["docs:largestChangestamp"]["@value"].front().Value().c_str() ); +} + } } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Syncer1.hh b/libgrive/src/drive/Syncer1.hh index 7f3d1cc4..b45e71c6 100644 --- a/libgrive/src/drive/Syncer1.hh +++ b/libgrive/src/drive/Syncer1.hh @@ -23,6 +23,8 @@ namespace gr { +class Feed; + namespace v1 { class Syncer1: public Syncer @@ -37,6 +39,11 @@ public : bool EditContent( Resource *res, bool new_rev ); bool Create( Resource *res ); + std::auto_ptr GetFolders(); + std::auto_ptr GetAll(); + std::auto_ptr GetChanges( long min_cstamp ); + long GetChangeStamp( long min_cstamp ); + private : bool Upload( Resource *res, const std::string& link, bool post); diff --git a/libgrive/src/drive2/CommonUri.hh b/libgrive/src/drive2/CommonUri.hh index 8f5304c1..34c6f04a 100644 --- a/libgrive/src/drive2/CommonUri.hh +++ b/libgrive/src/drive2/CommonUri.hh @@ -26,6 +26,7 @@ namespace gr { namespace v2 { namespace feeds { const std::string files = "https://www.googleapis.com/drive/v2/files" ; + const std::string changes = "https://www.googleapis.com/drive/v2/changes" ; } namespace mime_types @@ -33,9 +34,4 @@ namespace mime_types const std::string folder = "application/vnd.google-apps.folder" ; } -namespace kinds -{ - const std::string parent = "drive#parentReference" ; -} - } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index 14795231..321539fc 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -69,7 +69,9 @@ void Entry2::Update( const Val& item ) Val::Array parents = file["parents"].AsArray() ; for ( Val::Array::iterator i = parents.begin() ; i != parents.end() ; ++i ) - m_parent_hrefs.push_back( (*i)["parentLink"] ) ; // maybe .id? + { + m_parent_hrefs.push_back( (*i)["isRoot"].Bool() ? std::string() : (*i)["parentLink"] ) ; // maybe .id? + } // convert to lower case for easy comparison std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; diff --git a/libgrive/src/drive2/Feed.cc b/libgrive/src/drive2/Feed2.cc similarity index 60% rename from libgrive/src/drive2/Feed.cc rename to libgrive/src/drive2/Feed2.cc index 03fcea49..bb5abc4c 100644 --- a/libgrive/src/drive2/Feed.cc +++ b/libgrive/src/drive2/Feed2.cc @@ -17,42 +17,42 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -#include "Feed.hh" +#include "CommonUri.hh" +#include "Feed2.hh" +#include "Entry2.hh" #include "http/Agent.hh" #include "http/Header.hh" +#include "json/Val.hh" #include "json/ValResponse.hh" #include +#include namespace gr { namespace v2 { -Feed::Feed( ) +Feed2::Feed2( const std::string& url ): + Feed( url ) { } -// for example to find dirs: '?q=mimeType%3d%27' + mime_types::folder + '%27' -void Feed::Start( http::Agent *http, const std::string& url ) +bool Feed2::GetNext( http::Agent *http ) { + if ( m_next.empty() ) + return false ; + http::ValResponse out ; + http->Get( m_next, &out, http::Header() ) ; + Val m_content = out.Response() ; - http->Get( url, &out, http::Header() ) ; + Val::Array items = m_content["items"].AsArray() ; + m_entries.clear() ; + for ( Val::Array::iterator i = items.begin() ; i != items.end() ; ++i ) + m_entries.push_back( Entry2( *i ) ); - m_content = out.Response() ; -} - -bool Feed::GetNext( http::Agent *http ) -{ - assert( http != 0 ) ; - Val url ; - if ( m_content.Get( "nextLink", url ) ) - { - Start( http, url ) ; - return true ; - } - else - return false ; + m_next = m_content.Get( "nextLink", url ) ? url : std::string( "" ) ; + return true ; } -} } // end of namespace +} } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Feed.hh b/libgrive/src/drive2/Feed2.hh similarity index 81% rename from libgrive/src/drive2/Feed.hh rename to libgrive/src/drive2/Feed2.hh index 71917509..7948d206 100644 --- a/libgrive/src/drive2/Feed.hh +++ b/libgrive/src/drive2/Feed2.hh @@ -19,31 +19,19 @@ #pragma once -#include "json/Val.hh" +#include "base/Feed.hh" + #include "util/Exception.hh" #include -namespace gr -{ -namespace http -{ - class Agent ; -} - -class Val ; - -namespace v2 { +namespace gr { namespace v2 { -class Feed +class Feed2: public Feed { public : - Feed( ) ; - void Start( http::Agent *http, const std::string& url ) ; + Feed2( const std::string& url ) ; bool GetNext( http::Agent *http ) ; - -private : - Val m_content ; } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc new file mode 100644 index 00000000..1855e1c7 --- /dev/null +++ b/libgrive/src/drive2/Syncer2.cc @@ -0,0 +1,97 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "base/Resource.hh" +#include "CommonUri.hh" +#include "Entry2.hh" +#include "Feed2.hh" +#include "Syncer2.hh" + +#include "http/Agent.hh" +#include "http/Download.hh" +#include "http/Header.hh" +//#include "http/ResponseLog.hh" +#include "json/ValResponse.hh" + +#include "util/OS.hh" +#include "util/log/Log.hh" + +#include + +#include + +// for debugging +#include + +namespace gr { namespace v2 { + +Syncer2::Syncer2( http::Agent *http ): + Syncer( http ) +{ + assert( http != 0 ) ; +} + +void Syncer2::DeleteRemote( Resource *res ) +{ +} + +void Syncer2::Download( Resource *res, const fs::path& file ) +{ +} + +bool Syncer2::EditContent( Resource *res, bool new_rev ) +{ + return false ; +} + +bool Syncer2::Create( Resource *res ) +{ + return false ; +} + +std::auto_ptr Syncer2::GetFolders() +{ + return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers+and+mimeType%3d%27" + mime_types::folder + "%27" ) ); +} + +std::auto_ptr Syncer2::GetAll() +{ + return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers" ) ); +} + +std::string ChangesFeed( long changestamp, int maxResults = 1000 ) +{ + boost::format feed( feeds::changes + "?maxResults=%1%&includeSubscribed=false" + ( changestamp > 0 ? "&startChangeId=%2%" : "" ) ) ; + return ( changestamp > 0 ? feed % maxResults % changestamp : feed % maxResults ).str() ; +} + +std::auto_ptr Syncer2::GetChanges( long min_cstamp ) +{ + return std::auto_ptr( new Feed2( ChangesFeed( min_cstamp ) ) ); +} + +long Syncer2::GetChangeStamp( long min_cstamp ) +{ + http::ValResponse res ; + m_http->Get( ChangesFeed( min_cstamp, 1 ), &res, http::Header() ) ; + + return std::atoi( res.Response()["largestChangeId"].Str().c_str() ); +} + +} } // end of namespace gr::v1 diff --git a/libgrive/src/drive2/Syncer2.hh b/libgrive/src/drive2/Syncer2.hh new file mode 100644 index 00000000..35142a9c --- /dev/null +++ b/libgrive/src/drive2/Syncer2.hh @@ -0,0 +1,51 @@ +/* + grive: an GPL program to sync a local directory with Google Drive + Copyright (C) 2012 Wan Wai Ho + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "base/Syncer.hh" + +namespace gr { + +class Feed; + +namespace v2 { + +class Syncer2: public Syncer +{ + +public : + + Syncer2( http::Agent *http ); + + void DeleteRemote( Resource *res ); + void Download( Resource *res, const fs::path& file ); + bool EditContent( Resource *res, bool new_rev ); + bool Create( Resource *res ); + + std::auto_ptr GetFolders(); + std::auto_ptr GetAll(); + std::auto_ptr GetChanges( long min_cstamp ); + long GetChangeStamp( long min_cstamp ); + +private : + +} ; + +} } // end of namespace gr::v2 From a4521d9d62d292fe673fa9788533caffad67c216 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 16:59:58 +0300 Subject: [PATCH 078/166] Remove obsolete gdbm & iberty cmake modules --- README | 1 - cmake/Modules/FindGDBM.cmake | 48 ---------------------------------- cmake/Modules/FindIberty.cmake | 22 ---------------- libgrive/CMakeLists.txt | 10 ------- 4 files changed, 81 deletions(-) delete mode 100644 cmake/Modules/FindGDBM.cmake delete mode 100644 cmake/Modules/FindIberty.cmake diff --git a/README b/README index 7db8650d..3fb68b41 100644 --- a/README +++ b/README @@ -28,7 +28,6 @@ You need the following libraries: There are also some optional dependencies: - CppUnit (for unit tests) - libbfd (for backtrace) -- binutils (for libiberty, required for compilation in OpenSUSE & ubuntu) Grive uses cmake to build, see the instructions in: diff --git a/cmake/Modules/FindGDBM.cmake b/cmake/Modules/FindGDBM.cmake deleted file mode 100644 index e1ba00a2..00000000 --- a/cmake/Modules/FindGDBM.cmake +++ /dev/null @@ -1,48 +0,0 @@ -# - Find gdbm -# Find the native GDBM includes and library -# -# GDBM_INCLUDE_DIR - where to find gdbm.h, etc. -# GDBM_LIBRARIES - List of libraries when using gdbm. -# GDBM_FOUND - True if gdbm found. - - -IF (GDBM_INCLUDE_DIR) - # Already in cache, be silent - SET(GDBM_FIND_QUIETLY TRUE) -ENDIF (GDBM_INCLUDE_DIR) - -FIND_PATH(GDBM_INCLUDE_DIR gdbm.h - /usr/local/include - /usr/include - /opt/local/include -) - -SET(GDBM_NAMES gdbm) -FIND_LIBRARY(GDBM_LIBRARY - NAMES ${GDBM_NAMES} - PATHS /usr/lib /usr/local/lib /opt/local/lib -) - -IF (GDBM_INCLUDE_DIR AND GDBM_LIBRARY) - SET(GDBM_FOUND TRUE) - SET( GDBM_LIBRARIES ${GDBM_LIBRARY} ) -ELSE (GDBM_INCLUDE_DIR AND GDBM_LIBRARY) - SET(GDBM_FOUND FALSE) - SET( GDBM_LIBRARIES ) -ENDIF (GDBM_INCLUDE_DIR AND GDBM_LIBRARY) - -IF (GDBM_FOUND) - IF (NOT GDBM_FIND_QUIETLY) - MESSAGE(STATUS "Found GDBM: ${GDBM_LIBRARY}") - ENDIF (NOT GDBM_FIND_QUIETLY) -ELSE (GDBM_FOUND) - IF (GDBM_FIND_REQUIRED) - MESSAGE(STATUS "Looked for gdbm libraries named ${GDBMS_NAMES}.") - MESSAGE(FATAL_ERROR "Could NOT find gdbm library") - ENDIF (GDBM_FIND_REQUIRED) -ENDIF (GDBM_FOUND) - -MARK_AS_ADVANCED( - GDBM_LIBRARY - GDBM_INCLUDE_DIR -) diff --git a/cmake/Modules/FindIberty.cmake b/cmake/Modules/FindIberty.cmake deleted file mode 100644 index a14ba45b..00000000 --- a/cmake/Modules/FindIberty.cmake +++ /dev/null @@ -1,22 +0,0 @@ -# - Find Iberty -# This module finds libiberty. -# -# It sets the following variables: -# IBERTY_LIBRARY - The JSON-C library to link against. - -FIND_LIBRARY( IBERTY_LIBRARY NAMES iberty ) - -IF (IBERTY_LIBRARY) - - # show which JSON-C was found only if not quiet - MESSAGE( STATUS "Found libiberty: ${IBERTY_LIBRARY}") - - SET(IBERTY_FOUND TRUE) - -ELSE (IBERTY_LIBRARY) - - IF ( IBERTY_FIND_REQUIRED) - MESSAGE(FATAL_ERROR "Could not find libiberty. try to install binutil-devel?") - ENDIF (IBERTY_FIND_REQUIRED) - -ENDIF (IBERTY_LIBRARY) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 5a6558e2..95c1e559 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -8,7 +8,6 @@ find_package(EXPAT REQUIRED) find_package(Boost 1.40.0 COMPONENTS program_options filesystem unit_test_framework system REQUIRED) find_package(BFD) find_package(CppUnit) -find_package(Iberty) find_package(ZLIB) find_package(PkgConfig) @@ -29,12 +28,6 @@ if ( BFD_FOUND ) endif ( BFD_FOUND ) -if ( IBERTY_FOUND ) - set( OPT_LIBS ${OPT_LIBS} ${IBERTY_LIBRARY} ) -else ( IBERTY_FOUND ) - set( IBERTY_LIBRARY "" ) -endif ( IBERTY_FOUND ) - if ( ZLIB_FOUND ) set( OPT_LIBS ${OPT_LIBS} ${ZLIB_LIBRARIES} ) endif ( ZLIB_FOUND ) @@ -42,7 +35,6 @@ endif ( ZLIB_FOUND ) include_directories( ${libgrive_SOURCE_DIR}/src ${libgrive_SOURCE_DIR}/test - ${GDBM_INCLUDE_DIR} ${OPT_INCS} ${YAJL_INCLUDE_DIRS} ) @@ -87,9 +79,7 @@ target_link_libraries( grive ${YAJL_LIBRARIES} ${CURL_LIBRARIES} ${LIBGCRYPT_LIBRARIES} - ${GDBM_LIBRARIES} ${Boost_LIBRARIES} - ${IBERTY_LIBRARY} ${EXPAT_LIBRARY} ${OPT_LIBS} ) From 2d34d7708b4f8c2f3b2f8384395ba58aca728775 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 20:10:03 +0300 Subject: [PATCH 079/166] Implement upload methods for REST API Syncer (basic file upload to root now works) --- grive/src/main.cc | 5 +- libgrive/src/base/Syncer.cc | 17 +++++ libgrive/src/base/Syncer.hh | 2 +- libgrive/src/drive/Syncer1.cc | 17 +---- libgrive/src/drive/Syncer1.hh | 1 - libgrive/src/drive2/CommonUri.hh | 2 + libgrive/src/drive2/Entry2.cc | 10 +-- libgrive/src/drive2/Syncer2.cc | 115 +++++++++++++++++++++++++++++-- libgrive/src/drive2/Syncer2.hh | 3 +- libgrive/src/http/CurlAgent.cc | 7 +- libgrive/src/http/CurlAgent.hh | 2 +- libgrive/src/json/JsonParser.cc | 2 +- libgrive/src/json/JsonWriter.cc | 10 ++- libgrive/src/json/JsonWriter.hh | 3 + libgrive/src/json/Val.cc | 9 +++ 15 files changed, 165 insertions(+), 40 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index b323c056..f151e4b9 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -20,7 +20,7 @@ #include "util/Config.hh" #include "base/Drive.hh" -#include "drive/Syncer1.hh" +#include "drive2/Syncer2.hh" #include "http/CurlAgent.hh" #include "protocol/AuthAgent.hh" @@ -49,7 +49,6 @@ const std::string client_id = "22314510474.apps.googleusercontent.com" ; const std::string client_secret = "bl4ufi89h-9MkFlypcI7R785" ; using namespace gr ; -using namespace gr::v1 ; namespace po = boost::program_options; // libgcrypt insist this to be done in application, not library @@ -186,7 +185,7 @@ int Main( int argc, char **argv ) OAuth2 token( refresh_token, client_id, client_secret ) ; AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; - Syncer1 syncer( &agent ); + v2::Syncer2 syncer( &agent ); Drive drive( &syncer, config.GetAll() ) ; drive.DetectChanges() ; diff --git a/libgrive/src/base/Syncer.cc b/libgrive/src/base/Syncer.cc index 98d2acad..aa3b031f 100644 --- a/libgrive/src/base/Syncer.cc +++ b/libgrive/src/base/Syncer.cc @@ -21,6 +21,10 @@ #include "Resource.hh" #include "Entry.hh" #include "http/Agent.hh" +#include "http/Header.hh" +#include "http/Download.hh" +#include "util/OS.hh" +#include "util/log/Log.hh" namespace gr { @@ -34,6 +38,19 @@ http::Agent* Syncer::Agent() const return m_http; } +void Syncer::Download( Resource *res, const fs::path& file ) +{ + http::Download dl( file.string(), http::Download::NoChecksum() ) ; + long r = m_http->Get( res->ContentSrc(), &dl, http::Header() ) ; + if ( r <= 400 ) + { + if ( res->MTime() != DateTime() ) + os::SetFileTime( file, res->MTime() ) ; + else + Log( "encountered zero date time after downloading %1%", file, log::warning ) ; + } +} + void Syncer::AssignIDs( Resource *res, const Entry& remote ) { res->AssignIDs( remote ); diff --git a/libgrive/src/base/Syncer.hh b/libgrive/src/base/Syncer.hh index efe04bce..33375eee 100644 --- a/libgrive/src/base/Syncer.hh +++ b/libgrive/src/base/Syncer.hh @@ -50,7 +50,7 @@ public : http::Agent* Agent() const; virtual void DeleteRemote( Resource *res ) = 0; - virtual void Download( Resource *res, const fs::path& file ) = 0; + virtual void Download( Resource *res, const fs::path& file ); virtual bool EditContent( Resource *res, bool new_rev ) = 0; virtual bool Create( Resource *res ) = 0; diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index 3cde4172..d6d906d4 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -24,7 +24,6 @@ #include "Syncer1.hh" #include "http/Agent.hh" -#include "http/Download.hh" #include "http/Header.hh" //#include "http/ResponseLog.hh" #include "http/StringResponse.hh" @@ -35,6 +34,7 @@ #include "xml/String.hh" #include "xml/TreeBuilder.hh" +#include "util/File.hh" #include "util/OS.hh" #include "util/log/Log.hh" @@ -88,19 +88,6 @@ void Syncer1::DeleteRemote( Resource *res ) } } -void Syncer1::Download( Resource *res, const fs::path& file ) -{ - http::Download dl( file.string(), http::Download::NoChecksum() ) ; - long r = m_http->Get( res->ContentSrc(), &dl, http::Header() ) ; - if ( r <= 400 ) - { - if ( res->MTime() != DateTime() ) - os::SetFileTime( file, res->MTime() ) ; - else - Log( "encountered zero date time after downloading %1%", file, log::warning ) ; - } -} - bool Syncer1::EditContent( Resource *res, bool new_rev ) { assert( res->Parent() ) ; @@ -180,7 +167,7 @@ bool Syncer1::Upload( Resource *res, if ( retrying ) { file.Seek( 0, SEEK_SET ); - os::Sleep( 2 ); + os::Sleep( 5 ); } try diff --git a/libgrive/src/drive/Syncer1.hh b/libgrive/src/drive/Syncer1.hh index b45e71c6..924908b2 100644 --- a/libgrive/src/drive/Syncer1.hh +++ b/libgrive/src/drive/Syncer1.hh @@ -35,7 +35,6 @@ public : Syncer1( http::Agent *http ); void DeleteRemote( Resource *res ); - void Download( Resource *res, const fs::path& file ); bool EditContent( Resource *res, bool new_rev ); bool Create( Resource *res ); diff --git a/libgrive/src/drive2/CommonUri.hh b/libgrive/src/drive2/CommonUri.hh index 34c6f04a..6c2cee41 100644 --- a/libgrive/src/drive2/CommonUri.hh +++ b/libgrive/src/drive2/CommonUri.hh @@ -23,6 +23,8 @@ namespace gr { namespace v2 { +const std::string upload_base = "https://www.googleapis.com/upload/drive/v2/files" ; + namespace feeds { const std::string files = "https://www.googleapis.com/drive/v2/files" ; diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index 321539fc..ec1c3823 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -55,15 +55,17 @@ void Entry2::Update( const Val& item ) { m_title = file["title"] ; m_etag = file["etag"] ; - m_filename = file["originalFilename"] ; + Val fn; + m_filename = file.Get( "originalFilename", fn ) ? fn.Str() : std::string(); m_content_src = file["downloadUrl"] ; m_self_href = file["selfLink"] ; - m_mtime = DateTime( file["modificationDate"] ) ; + m_mtime = DateTime( file["modifiedDate"] ) ; - m_resource_id = file["id"]; // file#id ? + m_resource_id = file["id"]; m_md5 = file["md5Checksum"] ; - m_is_dir = file["mimeType"].Str() == v2::mime_types::folder ; + m_is_dir = file["mimeType"].Str() == mime_types::folder ; m_is_editable = file["editable"].Bool() ; + m_is_removed = file["labels"]["trashed"].Bool() ; m_parent_hrefs.clear( ) ; diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index 1855e1c7..f45ff7c0 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -26,8 +26,10 @@ #include "http/Agent.hh" #include "http/Download.hh" #include "http/Header.hh" +#include "http/StringResponse.hh" //#include "http/ResponseLog.hh" #include "json/ValResponse.hh" +#include "json/JsonWriter.hh" #include "util/OS.hh" #include "util/log/Log.hh" @@ -49,30 +51,129 @@ Syncer2::Syncer2( http::Agent *http ): void Syncer2::DeleteRemote( Resource *res ) { + http::StringResponse str ; + http::Header hdr ; + hdr.Add( "If-Match: " + res->ETag() ) ; + m_http->Post( res->SelfHref() + "/trash", "", &str, hdr ) ; } -void Syncer2::Download( Resource *res, const fs::path& file ) +bool Syncer2::EditContent( Resource *res, bool new_rev ) { + assert( res->Parent() ) ; + assert( !res->ResourceID().empty() ) ; + assert( res->Parent()->GetState() == Resource::sync ) ; + + if ( !res->IsEditable() ) + { + Log( "Cannot upload %1%: file read-only. %2%", res->Name(), res->StateStr(), log::warning ) ; + return false ; + } + + return Upload( res ) ; } -bool Syncer2::EditContent( Resource *res, bool new_rev ) +bool Syncer2::Create( Resource *res ) { - return false ; + assert( res->Parent() ) ; + assert( res->Parent()->IsFolder() ) ; + assert( res->Parent()->GetState() == Resource::sync ) ; + assert( res->ResourceID().empty() ) ; + + if ( !res->Parent()->IsEditable() ) + { + Log( "Cannot upload %1%: parent directory read-only. %2%", res->Name(), res->StateStr(), log::warning ) ; + return false ; + } + + return Upload( res ); } -bool Syncer2::Create( Resource *res ) +bool Syncer2::Upload( Resource *res ) { - return false ; + File file( res->Path() ) ; + std::ostringstream xcontent_len ; + xcontent_len << "Content-Length: " << file.Size() ; + + http::Header hdr ; + hdr.Add( "Content-Type: application/octet-stream" ) ; + hdr.Add( xcontent_len.str() ) ; + if ( !res->ETag().empty() ) + hdr.Add( "If-Match: " + res->ETag() ) ; + + Val meta; + meta.Add( "title", Val( res->Name() ) ); + if ( res->IsFolder() ) + { + meta.Add( "mimeType", Val( mime_types::folder ) ); + } + if ( !res->Parent()->IsRoot() ) + { + Val parent; + parent.Add( "id", Val( res->Parent()->ResourceID() ) ); + Val parents( Val::array_type ); + parents.Add( parent ); + meta.Add( "parents", parents ); + } + std::string json_meta = WriteJson( meta ); + + Val valr ; + + // Issue metadata update request + { + http::Header hdr2 ; + hdr2.Add( "Content-Type: application/json" ); + http::ValResponse vrsp ; + long http_code = 0; + if ( res->ResourceID().empty() ) + http_code = m_http->Post( feeds::files, json_meta, &vrsp, hdr2 ) ; + else + http_code = m_http->Put( feeds::files + "/" + res->ResourceID(), json_meta, &vrsp, hdr2 ) ; + valr = vrsp.Response(); + assert( !(valr["id"].Str().empty()) ); + } + + bool retrying = false; + while ( true ) + { + if ( retrying ) + { + file.Seek( 0, SEEK_SET ); + os::Sleep( 5 ); + } + + if ( !res->IsFolder() ) + { + http::ValResponse vrsp; + long http_code = m_http->Put( upload_base + "/" + valr["id"].Str() + "?uploadType=media", &file, &vrsp, hdr ) ; + if ( http_code == 410 || http_code == 412 ) + { + Log( "request failed with %1%, retrying whole upload in 5s", http_code, log::warning ) ; + retrying = true; + continue; + } + valr = vrsp.Response(); + assert( !(valr["id"].Str().empty()) ); + } + + if ( retrying ) + Log( "upload succeeded on retry", log::warning ); + Entry2 responseEntry = Entry2( valr ); + AssignIDs( res, responseEntry ) ; + AssignMTime( res, responseEntry.MTime() ); + break; + } + + return true ; } std::auto_ptr Syncer2::GetFolders() { - return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers+and+mimeType%3d%27" + mime_types::folder + "%27" ) ); + return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers+and+trashed%3dfalse+and+mimeType%3d%27" + mime_types::folder + "%27" ) ); } std::auto_ptr Syncer2::GetAll() { - return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers" ) ); + return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers+and+trashed%3dfalse" ) ); } std::string ChangesFeed( long changestamp, int maxResults = 1000 ) diff --git a/libgrive/src/drive2/Syncer2.hh b/libgrive/src/drive2/Syncer2.hh index 35142a9c..52da64c2 100644 --- a/libgrive/src/drive2/Syncer2.hh +++ b/libgrive/src/drive2/Syncer2.hh @@ -35,7 +35,6 @@ public : Syncer2( http::Agent *http ); void DeleteRemote( Resource *res ); - void Download( Resource *res, const fs::path& file ); bool EditContent( Resource *res, bool new_rev ); bool Create( Resource *res ); @@ -46,6 +45,8 @@ public : private : + bool Upload( Resource *res ); + } ; } } // end of namespace gr::v2 diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index d725e86c..53d1e271 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -232,18 +232,15 @@ long CurlAgent::Get( long CurlAgent::Post( const std::string& url, - const std::string& data, + const std::string& post_data, DataStream *dest, const Header& hdr ) { - Trace("HTTP POST \"%1%\" with \"%2%\"", url, data ) ; + Trace("HTTP POST \"%1%\" with \"%2%\"", url, post_data ) ; Init() ; CURL *curl = m_pimpl->curl ; - // make a copy because the parameter is const - std::string post_data = data ; - // set post specific options ::curl_easy_setopt(curl, CURLOPT_POST, 1L); ::curl_easy_setopt(curl, CURLOPT_POSTFIELDS, &post_data[0] ) ; diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index c142c3b1..7b4f2b72 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -43,7 +43,7 @@ public : long Put( const std::string& url, - const std::string& data, + const std::string& post_data, DataStream *dest, const Header& hdr ) ; diff --git a/libgrive/src/json/JsonParser.cc b/libgrive/src/json/JsonParser.cc index 50092cfa..9727b8e8 100644 --- a/libgrive/src/json/JsonParser.cc +++ b/libgrive/src/json/JsonParser.cc @@ -39,7 +39,7 @@ namespace int OnBool( void *ctx, int value ) { ValVisitor *b = reinterpret_cast(ctx) ; - b->Visit( static_cast(value) ) ; + b->Visit( static_cast(value) ) ; return true ; } diff --git a/libgrive/src/json/JsonWriter.cc b/libgrive/src/json/JsonWriter.cc index 9fbcc2bb..700c10c9 100644 --- a/libgrive/src/json/JsonWriter.cc +++ b/libgrive/src/json/JsonWriter.cc @@ -19,7 +19,7 @@ */ #include "JsonWriter.hh" -#include "util/DataStream.hh" +#include "util/StringStream.hh" #include @@ -108,4 +108,12 @@ void JsonWriter::WriteCallback( void *ctx, const char *str, std::size_t size ) pthis->m_impl->out->Write( str, size ) ; } +std::string WriteJson( const Val& val ) +{ + StringStream ss ; + JsonWriter wr( &ss ) ; + val.Visit( &wr ) ; + return ss.Str() ; +} + } // end of namespace diff --git a/libgrive/src/json/JsonWriter.hh b/libgrive/src/json/JsonWriter.hh index 350357d2..66f95bf9 100644 --- a/libgrive/src/json/JsonWriter.hh +++ b/libgrive/src/json/JsonWriter.hh @@ -20,6 +20,7 @@ #pragma once +#include "Val.hh" #include "ValVisitor.hh" #include @@ -53,5 +54,7 @@ private : std::auto_ptr m_impl ; } ; +std::string WriteJson( const Val& val ); + } // end of namespace diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index 31944ffe..1debddc2 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -114,11 +114,15 @@ Val::operator std::string() const int Val::Int() const { + if ( Type() == string_type ) + return std::atoi( As().c_str() ); return static_cast(As()) ; } double Val::Double() const { + if ( Type() == string_type ) + return std::atof( As().c_str() ); return As() ; } @@ -161,6 +165,11 @@ void Val::Add( const std::string& key, const Val& value ) As().insert( std::make_pair(key, value) ) ; } +void Val::Add( const Val& json ) +{ + As().push_back( json ) ; +} + void Val::Visit( ValVisitor *visitor ) const { switch ( Type() ) From 0bdbf875147cedbaa6fd426a2878cb45645c349f Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 21:05:24 +0300 Subject: [PATCH 080/166] Ignore local .trash, fix REST API - it mostly works --- libgrive/src/base/Resource.cc | 1 + libgrive/src/base/ResourceTree.cc | 2 +- libgrive/src/base/State.cc | 4 +--- libgrive/src/drive/Entry1.cc | 4 ++-- libgrive/src/drive2/Entry2.cc | 14 ++++++++------ 5 files changed, 13 insertions(+), 12 deletions(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 889e0706..3f126c85 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -42,6 +42,7 @@ Resource::Resource( const fs::path& root_folder ) : m_name ( root_folder.string() ), m_kind ( "folder" ), m_id ( "folder:root" ), + m_href ( "root" ), m_parent ( 0 ), m_state ( sync ), m_is_editable( true ) diff --git a/libgrive/src/base/ResourceTree.cc b/libgrive/src/base/ResourceTree.cc index 3aa8d552..738b8028 100644 --- a/libgrive/src/base/ResourceTree.cc +++ b/libgrive/src/base/ResourceTree.cc @@ -30,7 +30,7 @@ namespace gr { using namespace details ; ResourceTree::ResourceTree( const fs::path& rootFolder ) : - m_root( new Resource( rootFolder ) ) + m_root( new Resource( rootFolder ) ) { m_set.insert( m_root ) ; } diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index d0525f2b..4aaaf220 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -61,7 +61,7 @@ void State::FromLocal( const fs::path& p ) bool State::IsIgnore( const std::string& filename ) { - return filename == ".grive" || filename == ".grive_state"; + return filename == ".grive" || filename == ".grive_state" || filename == ".trash"; } void State::FromLocal( const fs::path& p, Resource* folder ) @@ -135,9 +135,7 @@ void State::FromRemote( const Entry& e ) FromChange( e ) ; else if ( !Update( e ) ) - { m_unresolved.push_back( e ) ; - } } void State::ResolveEntry() diff --git a/libgrive/src/drive/Entry1.cc b/libgrive/src/drive/Entry1.cc index d44b421d..a7a3dd94 100644 --- a/libgrive/src/drive/Entry1.cc +++ b/libgrive/src/drive/Entry1.cc @@ -72,8 +72,8 @@ void Entry1::Update( const xml::Node& n ) for ( xml::NodeSet::iterator i = parents.begin() ; i != parents.end() ; ++i ) { std::string href = (*i)["@href"]; - if ( href == root_href ) // API-independent root href is empty! - href = ""; + if ( href == root_href ) + href = "root"; // API-independent root href m_parent_hrefs.push_back( href ) ; } diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index ec1c3823..f0dfc133 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -57,26 +57,28 @@ void Entry2::Update( const Val& item ) m_etag = file["etag"] ; Val fn; m_filename = file.Get( "originalFilename", fn ) ? fn.Str() : std::string(); - m_content_src = file["downloadUrl"] ; m_self_href = file["selfLink"] ; m_mtime = DateTime( file["modifiedDate"] ) ; m_resource_id = file["id"]; - m_md5 = file["md5Checksum"] ; m_is_dir = file["mimeType"].Str() == mime_types::folder ; m_is_editable = file["editable"].Bool() ; m_is_removed = file["labels"]["trashed"].Bool() ; + if ( !m_is_dir ) + { + m_md5 = file["md5Checksum"] ; + m_content_src = file["downloadUrl"] ; + // convert to lower case for easy comparison + std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; + } m_parent_hrefs.clear( ) ; Val::Array parents = file["parents"].AsArray() ; for ( Val::Array::iterator i = parents.begin() ; i != parents.end() ; ++i ) { - m_parent_hrefs.push_back( (*i)["isRoot"].Bool() ? std::string() : (*i)["parentLink"] ) ; // maybe .id? + m_parent_hrefs.push_back( (*i)["isRoot"].Bool() ? std::string( "root" ) : (*i)["parentLink"] ) ; } - - // convert to lower case for easy comparison - std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; } } From 29241002eb842140df91fde7ec297afed644edd2 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 17 May 2015 22:09:42 +0300 Subject: [PATCH 081/166] Add myself to copyrights, change README --- CMakeLists.txt | 2 +- README | 61 ------------------------ README.md | 80 ++++++++++++++++++++++++++++++++ libgrive/src/drive/CommonUri.hh | 2 +- libgrive/src/drive/Entry1.cc | 4 +- libgrive/src/drive/Entry1.hh | 4 +- libgrive/src/drive/Feed1.cc | 4 +- libgrive/src/drive/Feed1.hh | 4 +- libgrive/src/drive/Syncer1.cc | 4 +- libgrive/src/drive/Syncer1.hh | 4 +- libgrive/src/drive2/CommonUri.hh | 4 +- libgrive/src/drive2/Entry2.cc | 4 +- libgrive/src/drive2/Entry2.hh | 4 +- libgrive/src/drive2/Feed2.cc | 4 +- libgrive/src/drive2/Feed2.hh | 4 +- libgrive/src/drive2/Syncer2.cc | 4 +- libgrive/src/drive2/Syncer2.hh | 4 +- 17 files changed, 108 insertions(+), 89 deletions(-) delete mode 100644 README create mode 100644 README.md diff --git a/CMakeLists.txt b/CMakeLists.txt index def7ca00..293a08bd 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! -set( GRIVE_VERSION "0.3.0-pre" ) +set( GRIVE_VERSION "0.4.0-pre" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) diff --git a/README b/README deleted file mode 100644 index 3fb68b41..00000000 --- a/README +++ /dev/null @@ -1,61 +0,0 @@ -Grive 0.3.0 -25 July 2012 - -http://www.lbreda.com/grive/ - -Grive can be considered still beta quality. It simply downloads all the files in your -Google Drive into the current directory. After you make some changes to the local files, run -grive again and it will upload your changes back to your Google Drive. New files created locally -or in Google Drive will be uploaded or downloaded respectively. Deleted files will also be "removed". -Currently Grive will NOT destroy any of your files: it will only move the files to a -directory named .trash or put them in the Google Drive trash. You can always recover them. - -There are a few things that Grive does not do at the moment: -- wait for changes in file system to occur and upload. A sync is only performed when you run Grive. -- symbolic links support -- support for Google documents - -Of course these will be added in the future, possibly the next release. - -You need the following libraries: - -- yajl -- libcurl -- libstdc++ -- libgcrypt -- Boost (Boost filesystem and program_option are required) - -There are also some optional dependencies: -- CppUnit (for unit tests) -- libbfd (for backtrace) - -Grive uses cmake to build, see the instructions in: - -http://www.lbreda.com/grive/installation - -...for detailed procedures to compile Grive. - -When Grive is run for the first time, you should use the "-a" argument to grant -permission to Grive to access to your Google Drive. A URL should be printed. -Go to the link. You will need to login to your Google account if you haven't -done so. After granting the permission to Grive, the browser will show you -an authenication code. Copy-and-paste that to the standard input of Grive. - -If everything works fine, Grive will create .grive and .grive_state files in your -current directory. It will also start downloading files from your Google Drive to -your current directory. - -Enjoy! - -Version History: - -Grive v0.3: Bug fix & minor feature release -Fixed bugs: - #93: missing reference count increment in one of the Json constructors - #82: retry for HTTP error 500 & 503 - #77: Fixed a bug where grive crashed on the first run. - -New features: - #87: support for revisions - #86: partial sync (contributed by justin at tierramedia.com) - \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 00000000..9736430a --- /dev/null +++ b/README.md @@ -0,0 +1,80 @@ +# Grive2 0.4.0 + +17 May 2015, Vitaliy Filippov + +http://yourcmc.ru/wiki/Grive2 + +This is the fork of original "Grive" (https://github.com/Grive/grive) Google Drive client +with the support for the new Drive REST API and partial sync. + +Grive can be considered still beta or pre-beta quality. It simply downloads all the files in your +Google Drive into the current directory. After you make some changes to the local files, run +grive again and it will upload your changes back to your Google Drive. New files created locally +or in Google Drive will be uploaded or downloaded respectively. Deleted files will also be "removed". +Currently Grive will NOT destroy any of your files: it will only move the files to a +directory named .trash or put them in the Google Drive trash. You can always recover them. + +There are a few things that Grive does not do at the moment: +- continously wait for changes in file system or in Google Drive to occur and upload. + A sync is only performed when you run Grive, and it calculates checksums for all files every time. +- symbolic links support. +- support for Google documents. + +These may be added in the future, possibly the next release. + +You need the following libraries: + +- yajl +- libcurl +- libstdc++ +- libgcrypt +- Boost (Boost filesystem and program_option are required) + +There are also some optional dependencies: +- CppUnit (for unit tests) +- libbfd (for backtrace) + +Grive uses cmake to build. Basic install sequence is + + mkdir build + cd build + cmake .. + make -j4 + sudo make install + +For the detailed instructions, see http://yourcmc.ru/wiki/Grive2#Installation + +When Grive is run for the first time, you should use the "-a" argument to grant +permission to Grive to access to your Google Drive. A URL should be printed. +Go to the link. You will need to login to your Google account if you haven't +done so. After granting the permission to Grive, the browser will show you +an authenication code. Copy-and-paste that to the standard input of Grive. + +If everything works fine, Grive will create .grive and .grive_state files in your +current directory. It will also start downloading files from your Google Drive to +your current directory. + +Enjoy! + +## Version History + +### Grive2 v0.4 + +First fork release, by Vitaliy Filippov / vitalif at mail*ru +- Support for the new Google Drive REST API (old "Document List" API is shut down by Google 20 April 2015) +- REAL support for partial sync: syncs only one subdirectory with `grive -d subdir` +- Major refactoring - a lot of dead code removed, JSON-C is not used anymore, API-specific code is split from non-API-specific +- Some stability fixes from Visa Putkinen https://github.com/visap/grive/commits/visa +- Slightly reduce number of syscalls when reading local files. + +### Grive v0.3 + +Bug fix & minor feature release. Fixed bugs: +- #93: missing reference count increment in one of the Json constructors +- #82: retry for HTTP error 500 & 503 +- #77: Fixed a bug where grive crashed on the first run. + +New features: +- #87: support for revisions +- #86: ~~partial sync (contributed by justin at tierramedia.com)~~ that's not partial sync, + that's only support for specifying local path on command line diff --git a/libgrive/src/drive/CommonUri.hh b/libgrive/src/drive/CommonUri.hh index 98cc9594..bbc3b0b6 100644 --- a/libgrive/src/drive/CommonUri.hh +++ b/libgrive/src/drive/CommonUri.hh @@ -1,5 +1,5 @@ /* - grive: an GPL program to sync a local directory with Google Drive + Common URIs for the old "Document List" Google Docs API Copyright (C) 2012 Wan Wai Ho This program is free software; you can redistribute it and/or diff --git a/libgrive/src/drive/Entry1.cc b/libgrive/src/drive/Entry1.cc index a7a3dd94..9f75138a 100644 --- a/libgrive/src/drive/Entry1.cc +++ b/libgrive/src/drive/Entry1.cc @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + Item class implementation for the old "Document List" Google Docs API + Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive/Entry1.hh b/libgrive/src/drive/Entry1.hh index b606f8af..6fff374e 100644 --- a/libgrive/src/drive/Entry1.hh +++ b/libgrive/src/drive/Entry1.hh @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + Item class implementation for the old "Document List" Google Docs API + Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive/Feed1.cc b/libgrive/src/drive/Feed1.cc index 05093aaa..addde4bb 100644 --- a/libgrive/src/drive/Feed1.cc +++ b/libgrive/src/drive/Feed1.cc @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + Item list ("Feed") implementation for the old "Document List" Google Docs API + Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive/Feed1.hh b/libgrive/src/drive/Feed1.hh index 84466933..40f88a8f 100644 --- a/libgrive/src/drive/Feed1.hh +++ b/libgrive/src/drive/Feed1.hh @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + Item list ("Feed") implementation for the old "Document List" Google Docs API + Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index d6d906d4..0ca7e153 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + Syncer implementation for the old "Document List" Google Docs API + Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive/Syncer1.hh b/libgrive/src/drive/Syncer1.hh index 924908b2..d11c48b5 100644 --- a/libgrive/src/drive/Syncer1.hh +++ b/libgrive/src/drive/Syncer1.hh @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + Syncer implementation for the old "Document List" Google Docs API + Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive2/CommonUri.hh b/libgrive/src/drive2/CommonUri.hh index 6c2cee41..9cef07b6 100644 --- a/libgrive/src/drive2/CommonUri.hh +++ b/libgrive/src/drive2/CommonUri.hh @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + Common URIs for REST API + Copyright (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index f0dfc133..73f11a36 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + REST API item class implementation + Copyright (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive2/Entry2.hh b/libgrive/src/drive2/Entry2.hh index 624df30a..2c848d2c 100644 --- a/libgrive/src/drive2/Entry2.hh +++ b/libgrive/src/drive2/Entry2.hh @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + REST API item class implementation + Copyright (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive2/Feed2.cc b/libgrive/src/drive2/Feed2.cc index bb5abc4c..8a3244ee 100644 --- a/libgrive/src/drive2/Feed2.cc +++ b/libgrive/src/drive2/Feed2.cc @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + REST API item list ("Feed") implementation + Copyright (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive2/Feed2.hh b/libgrive/src/drive2/Feed2.hh index 7948d206..4de411e9 100644 --- a/libgrive/src/drive2/Feed2.hh +++ b/libgrive/src/drive2/Feed2.hh @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + REST API item list ("Feed") implementation + Copyright (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index f45ff7c0..b08ad4c2 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + REST API Syncer implementation + Copyright (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License diff --git a/libgrive/src/drive2/Syncer2.hh b/libgrive/src/drive2/Syncer2.hh index 52da64c2..0ab6286c 100644 --- a/libgrive/src/drive2/Syncer2.hh +++ b/libgrive/src/drive2/Syncer2.hh @@ -1,6 +1,6 @@ /* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho + REST API Syncer implementation + Copyright (C) 2015 Vitaliy Filippov This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License From c006e75d26c08176d336df9ebcf7c915a3e0adb9 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 20 May 2015 02:10:04 +0300 Subject: [PATCH 082/166] Return libiberty back --- README.md | 1 + cmake/Modules/FindIberty.cmake | 22 ++++++++++++++++++++++ libgrive/CMakeLists.txt | 8 ++++++++ 3 files changed, 31 insertions(+) create mode 100644 cmake/Modules/FindIberty.cmake diff --git a/README.md b/README.md index 9736430a..146828c7 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ You need the following libraries: There are also some optional dependencies: - CppUnit (for unit tests) - libbfd (for backtrace) +- binutils (for libiberty, required for compilation in OpenSUSE, Ubuntu, Arch and etc) Grive uses cmake to build. Basic install sequence is diff --git a/cmake/Modules/FindIberty.cmake b/cmake/Modules/FindIberty.cmake new file mode 100644 index 00000000..6562419a --- /dev/null +++ b/cmake/Modules/FindIberty.cmake @@ -0,0 +1,22 @@ +# - Find Iberty +# This module finds libiberty. +# +# It sets the following variables: +# IBERTY_LIBRARY - The library to link against. + +FIND_LIBRARY( IBERTY_LIBRARY NAMES iberty ) + +IF (IBERTY_LIBRARY) + + # show which library was found only if not quiet + MESSAGE( STATUS "Found libiberty: ${IBERTY_LIBRARY}") + + SET(IBERTY_FOUND TRUE) + +ELSE (IBERTY_LIBRARY) + + IF ( IBERTY_FIND_REQUIRED) + MESSAGE(FATAL_ERROR "Could not find libiberty. try to install binutil-devel?") + ENDIF (IBERTY_FIND_REQUIRED) + +ENDIF (IBERTY_LIBRARY) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 95c1e559..7fadd8dc 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -8,6 +8,7 @@ find_package(EXPAT REQUIRED) find_package(Boost 1.40.0 COMPONENTS program_options filesystem unit_test_framework system REQUIRED) find_package(BFD) find_package(CppUnit) +find_package(Iberty) find_package(ZLIB) find_package(PkgConfig) @@ -28,6 +29,12 @@ if ( BFD_FOUND ) endif ( BFD_FOUND ) +if ( IBERTY_FOUND ) + set( OPT_LIBS ${OPT_LIBS} ${IBERTY_LIBRARY} ) +else ( IBERTY_FOUND ) + set( IBERTY_LIBRARY "" ) +endif ( IBERTY_FOUND ) + if ( ZLIB_FOUND ) set( OPT_LIBS ${OPT_LIBS} ${ZLIB_LIBRARIES} ) endif ( ZLIB_FOUND ) @@ -80,6 +87,7 @@ target_link_libraries( grive ${CURL_LIBRARIES} ${LIBGCRYPT_LIBRARIES} ${Boost_LIBRARIES} + ${IBERTY_LIBRARY} ${EXPAT_LIBRARY} ${OPT_LIBS} ) From 815d0ea44ea49044106cbc2b6685b57f078b5e9a Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 20 May 2015 02:07:43 +0300 Subject: [PATCH 083/166] Make libbfd happy. In conjunction with libiberty fixes #1 --- libgrive/src/bfd/SymbolInfo.cc | 1 + 1 file changed, 1 insertion(+) diff --git a/libgrive/src/bfd/SymbolInfo.cc b/libgrive/src/bfd/SymbolInfo.cc index afd3ba14..5876cc00 100644 --- a/libgrive/src/bfd/SymbolInfo.cc +++ b/libgrive/src/bfd/SymbolInfo.cc @@ -22,6 +22,7 @@ #include +#define PACKAGE "libgrive" #include #include #include From 37b2cf3102ab453d5bd53f832bcf4230ec73f1b9 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 20 May 2015 15:14:24 +0300 Subject: [PATCH 084/166] Ignore files without md5Checksum. Fixes #2 --- libgrive/src/drive2/Entry2.cc | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index 73f11a36..e5db8f06 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -66,10 +66,19 @@ void Entry2::Update( const Val& item ) m_is_removed = file["labels"]["trashed"].Bool() ; if ( !m_is_dir ) { - m_md5 = file["md5Checksum"] ; - m_content_src = file["downloadUrl"] ; - // convert to lower case for easy comparison - std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; + if ( !file.Has( "md5Checksum" ) ) + { + // This is either a google docs document or a not-yet-uploaded file. Ignore it. + // FIXME: We'll need to compare timestamps to support google docs. + m_is_removed = true; + } + else + { + m_md5 = file["md5Checksum"] ; + m_content_src = file["downloadUrl"] ; + // convert to lower case for easy comparison + std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; + } } m_parent_hrefs.clear( ) ; From 82705f39b0ae921c6abb9f86cad75e9a53187b44 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 21 May 2015 19:20:58 +0300 Subject: [PATCH 085/166] Fix 412's on reupload. Fixes #5 --- libgrive/src/base/Resource.cc | 1 + libgrive/src/drive2/Syncer2.cc | 55 +++++++++++++++------------------- 2 files changed, 25 insertions(+), 31 deletions(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 3f126c85..058f08f8 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -386,6 +386,7 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) case local_new : Log( "sync %1% doesn't exist in server, uploading", path, log::info ) ; + // FIXME: (?) do not write new timestamp on failed upload if ( syncer && syncer->Create( this ) ) m_state = sync ; break ; diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index b08ad4c2..0b38a5a0 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -90,16 +90,6 @@ bool Syncer2::Create( Resource *res ) bool Syncer2::Upload( Resource *res ) { - File file( res->Path() ) ; - std::ostringstream xcontent_len ; - xcontent_len << "Content-Length: " << file.Size() ; - - http::Header hdr ; - hdr.Add( "Content-Type: application/octet-stream" ) ; - hdr.Add( xcontent_len.str() ) ; - if ( !res->ETag().empty() ) - hdr.Add( "If-Match: " + res->ETag() ) ; - Val meta; meta.Add( "title", Val( res->Name() ) ); if ( res->IsFolder() ) @@ -129,40 +119,43 @@ bool Syncer2::Upload( Resource *res ) else http_code = m_http->Put( feeds::files + "/" + res->ResourceID(), json_meta, &vrsp, hdr2 ) ; valr = vrsp.Response(); - assert( !(valr["id"].Str().empty()) ); + assert( !( valr["id"].Str().empty() ) ); } - bool retrying = false; - while ( true ) + if ( !res->IsFolder() ) { - if ( retrying ) + while ( true ) { - file.Seek( 0, SEEK_SET ); - os::Sleep( 5 ); - } + File file( res->Path() ) ; + std::ostringstream xcontent_len ; + xcontent_len << "Content-Length: " << file.Size() ; + + http::Header hdr ; + hdr.Add( "Content-Type: application/octet-stream" ) ; + hdr.Add( xcontent_len.str() ) ; + if ( valr.Has( "etag" ) ) + hdr.Add( "If-Match: " + valr["etag"].Str() ) ; - if ( !res->IsFolder() ) - { http::ValResponse vrsp; long http_code = m_http->Put( upload_base + "/" + valr["id"].Str() + "?uploadType=media", &file, &vrsp, hdr ) ; if ( http_code == 410 || http_code == 412 ) { Log( "request failed with %1%, retrying whole upload in 5s", http_code, log::warning ) ; - retrying = true; - continue; + os::Sleep( 5 ); + } + else + { + valr = vrsp.Response() ; + assert( !( valr["id"].Str().empty() ) ); + break ; } - valr = vrsp.Response(); - assert( !(valr["id"].Str().empty()) ); } - - if ( retrying ) - Log( "upload succeeded on retry", log::warning ); - Entry2 responseEntry = Entry2( valr ); - AssignIDs( res, responseEntry ) ; - AssignMTime( res, responseEntry.MTime() ); - break; } - + + Entry2 responseEntry = Entry2( valr ) ; + AssignIDs( res, responseEntry ) ; + AssignMTime( res, responseEntry.MTime() ) ; + return true ; } From 06eb1a7df2032144f99473fd812412514834a1ee Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 31 May 2015 22:35:15 +0300 Subject: [PATCH 086/166] Fix FreeBSD build --- grive/CMakeLists.txt | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/grive/CMakeLists.txt b/grive/CMakeLists.txt index d5f8ebef..837a36d5 100644 --- a/grive/CMakeLists.txt +++ b/grive/CMakeLists.txt @@ -5,10 +5,11 @@ find_package(Boost COMPONENTS program_options REQUIRED) include_directories( ${grive_SOURCE_DIR}/../libgrive/src ${OPT_INCS} + ${Boost_INCLUDE_DIRS} ) file (GLOB GRIVE_EXE_SRC - ${grive_SOURCE_DIR}/src/*.cc + ${grive_SOURCE_DIR}/src/*.cc ) add_executable( grive_executable @@ -17,11 +18,11 @@ add_executable( grive_executable target_link_libraries( grive_executable ${Boost_LIBRARIES} - grive + grive ) set_target_properties( grive_executable - PROPERTIES OUTPUT_NAME grive + PROPERTIES OUTPUT_NAME grive ) install(TARGETS grive_executable RUNTIME DESTINATION bin) From 70ec926e2a6ded8aa3cdb82f8fa8a0d0b528f4db Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 27 Jun 2015 22:58:01 +0300 Subject: [PATCH 087/166] Add response body and headers to diagnostic information on HTTP 400 and above Also should fix #13 because stops AuthAgent from using the same response multiple times --- libgrive/src/drive/Syncer1.cc | 2 +- libgrive/src/drive2/Syncer2.cc | 2 +- libgrive/src/http/Agent.hh | 3 ++ libgrive/src/http/CurlAgent.cc | 59 +++++++++++++++++++++++------- libgrive/src/http/CurlAgent.hh | 5 ++- libgrive/src/http/Error.hh | 17 +++++---- libgrive/src/protocol/AuthAgent.cc | 36 +++++++++++------- libgrive/src/protocol/AuthAgent.hh | 3 ++ 8 files changed, 90 insertions(+), 37 deletions(-) diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index 0ca7e153..31a331a0 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -225,7 +225,7 @@ bool Syncer1::Upload( Resource *res, if ( http_code == 410 || http_code == 412 ) { - Log( "request failed with %1%, retrying whole upload in 5s", http_code, log::warning ) ; + Log( "request failed with %1%, body: %2%, retrying whole upload in 5s", http_code, m_http->LastError(), log::warning ) ; retrying = true; continue; } diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index 0b38a5a0..d0f6f3c7 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -140,7 +140,7 @@ bool Syncer2::Upload( Resource *res ) long http_code = m_http->Put( upload_base + "/" + valr["id"].Str() + "?uploadType=media", &file, &vrsp, hdr ) ; if ( http_code == 410 || http_code == 412 ) { - Log( "request failed with %1%, retrying whole upload in 5s", http_code, log::warning ) ; + Log( "request failed with %1%, body: %2%. retrying whole upload in 5s", http_code, m_http->LastError(), log::warning ) ; os::Sleep( 5 ); } else diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index a1903ce9..5f4d5dec 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -64,6 +64,9 @@ public : DataStream *dest, const Header& hdr ) = 0 ; + virtual std::string LastError() const = 0 ; + virtual std::string LastErrorHeaders() const = 0 ; + virtual std::string RedirLocation() const = 0 ; virtual std::string Escape( const std::string& str ) = 0 ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 53d1e271..bc766b77 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -85,6 +85,10 @@ struct CurlAgent::Impl { CURL *curl ; std::string location ; + bool error ; + std::string error_headers ; + std::string error_data ; + DataStream *dest ; } ; CurlAgent::CurlAgent() : @@ -96,11 +100,15 @@ CurlAgent::CurlAgent() : void CurlAgent::Init() { ::curl_easy_reset( m_pimpl->curl ) ; - ::curl_easy_setopt( m_pimpl->curl, CURLOPT_SSL_VERIFYPEER, 0L ) ; + ::curl_easy_setopt( m_pimpl->curl, CURLOPT_SSL_VERIFYPEER, 0L ) ; ::curl_easy_setopt( m_pimpl->curl, CURLOPT_SSL_VERIFYHOST, 0L ) ; ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADERFUNCTION, &CurlAgent::HeaderCallback ) ; - ::curl_easy_setopt( m_pimpl->curl, CURLOPT_WRITEHEADER , this ) ; - ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADER, 0L ) ; + ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADERDATA, this ) ; + ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADER, 0L ) ; + m_pimpl->error = false; + m_pimpl->error_headers = ""; + m_pimpl->error_data = ""; + m_pimpl->dest = NULL; } CurlAgent::~CurlAgent() @@ -110,9 +118,16 @@ CurlAgent::~CurlAgent() std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) { - char *str = reinterpret_cast(ptr) ; + char *str = static_cast(ptr) ; std::string line( str, str + size*nmemb ) ; + // Check for error (HTTP 400 and above) + if ( line.substr( 0, 5 ) == "HTTP/" && line[9] >= '4' ) + pthis->m_pimpl->error = true; + + if ( pthis->m_pimpl->error ) + pthis->m_pimpl->error_headers += line; + static const std::string loc = "Location: " ; std::size_t pos = line.find( loc ) ; if ( pos != line.npos ) @@ -124,10 +139,16 @@ std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, Cur return size*nmemb ; } -std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, DataStream *recv ) +std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, CurlAgent *pthis ) { - assert( recv != 0 ) ; - return recv->Write( static_cast(ptr), size * nmemb ) ; + assert( pthis != 0 ) ; + if ( pthis->m_pimpl->error && pthis->m_pimpl->error_data.size() < 65536 ) + { + // Do not feed error responses to destination stream + pthis->m_pimpl->error_data.append( static_cast(ptr), size * nmemb ) ; + return size * nmemb ; + } + return pthis->m_pimpl->dest->Write( static_cast(ptr), size * nmemb ) ; } long CurlAgent::ExecCurl( @@ -142,21 +163,23 @@ long CurlAgent::ExecCurl( ::curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, error ) ; ::curl_easy_setopt(curl, CURLOPT_URL, url.c_str()); ::curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlAgent::Receive ) ; - ::curl_easy_setopt(curl, CURLOPT_WRITEDATA, dest ) ; + ::curl_easy_setopt(curl, CURLOPT_WRITEDATA, this ) ; + m_pimpl->dest = dest ; SetHeader( hdr ) ; -// dest->Clear() ; CURLcode curl_code = ::curl_easy_perform(curl); // get the HTTP response code long http_code = 0; ::curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code); Trace( "HTTP response %1%", http_code ) ; - - // reset the curl buffer to prevent it from touch our "error" buffer + + // reset the curl buffer to prevent it from touching our "error" buffer ::curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, 0 ) ; - + + m_pimpl->dest = NULL; + // only throw for libcurl errors if ( curl_code != CURLE_OK ) { @@ -165,7 +188,7 @@ long CurlAgent::ExecCurl( << CurlCode( curl_code ) << Url( url ) << CurlErrMsg( error ) - << HttpHeader( hdr ) + << HttpRequestHeaders( hdr ) ) ; } @@ -275,6 +298,16 @@ void CurlAgent::SetHeader( const Header& hdr ) ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HTTPHEADER, curl_hdr ) ; } +std::string CurlAgent::LastError() const +{ + return m_pimpl->error_data ; +} + +std::string CurlAgent::LastErrorHeaders() const +{ + return m_pimpl->error_headers ; +} + std::string CurlAgent::RedirLocation() const { return m_pimpl->location ; diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index 7b4f2b72..b7125336 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -70,6 +70,9 @@ public : DataStream *dest, const Header& hdr ) ; + std::string LastError() const ; + std::string LastErrorHeaders() const ; + std::string RedirLocation() const ; std::string Escape( const std::string& str ) ; @@ -77,7 +80,7 @@ public : private : static std::size_t HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; - static std::size_t Receive( void* ptr, size_t size, size_t nmemb, DataStream *recv ) ; + static std::size_t Receive( void* ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; void SetHeader( const Header& hdr ) ; long ExecCurl( diff --git a/libgrive/src/http/Error.hh b/libgrive/src/http/Error.hh index 815ee04f..897b4555 100644 --- a/libgrive/src/http/Error.hh +++ b/libgrive/src/http/Error.hh @@ -27,21 +27,24 @@ namespace gr { namespace http { struct Error : virtual Exception {} ; // CURL error code -typedef boost::error_info CurlCode ; +typedef boost::error_info CurlCode ; // CURL error message -typedef boost::error_info CurlErrMsg ; +typedef boost::error_info CurlErrMsg ; // URL -typedef boost::error_info Url ; +typedef boost::error_info Url ; -// HTTP headers -typedef boost::error_info HttpHeader ; +// HTTP request headers +typedef boost::error_info HttpRequestHeaders ; // HTTP response code -typedef boost::error_info HttpResponse ; +typedef boost::error_info HttpResponseCode ; + +// HTTP response headers +typedef boost::error_info HttpResponseHeaders ; // HTTP response body -typedef boost::error_info HttpResponseText ; +typedef boost::error_info HttpResponseText ; } } // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 30a97223..e24d4e02 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -21,7 +21,6 @@ #include "http/Error.hh" #include "http/Header.hh" -#include "http/XmlResponse.hh" #include "util/log/Log.hh" #include "util/OS.hh" #include "util/File.hh" @@ -77,9 +76,6 @@ long AuthAgent::Put( keepTrying = CheckRetry( response ); if ( keepTrying ) { file->Seek( 0, SEEK_SET ); - XmlResponse *xmlResponse = dynamic_cast(dest); - if( xmlResponse ) - xmlResponse->Clear(); } } @@ -135,6 +131,16 @@ long AuthAgent::Custom( return CheckHttpResponse(response, url, auth) ; } +std::string AuthAgent::LastError() const +{ + return m_agent->LastError() ; +} + +std::string AuthAgent::LastErrorHeaders() const +{ + return m_agent->LastErrorHeaders() ; +} + std::string AuthAgent::RedirLocation() const { return m_agent->RedirLocation() ; @@ -152,11 +158,11 @@ std::string AuthAgent::Unescape( const std::string& str ) bool AuthAgent::CheckRetry( long response ) { - // HTTP 500 and 503 should be temperory. just wait a bit and retry + // HTTP 500 and 503 should be temporary. just wait a bit and retry if ( response == 500 || response == 503 ) { - Log( "resquest failed due to temperory error: %1%. retrying in 5 seconds", - response, log::warning ) ; + Log( "request failed due to temporary error: %1%, body: %2%. retrying in 5 seconds", + response, m_agent->LastError(), log::warning ) ; os::Sleep( 5 ) ; return true ; @@ -165,7 +171,7 @@ bool AuthAgent::CheckRetry( long response ) // HTTP 401 Unauthorized. the auth token has been expired. refresh it else if ( response == 401 ) { - Log( "resquest failed due to auth token expired: %1%. refreshing token", + Log( "request failed due to auth token expired: %1%. refreshing token", response, log::warning ) ; os::Sleep( 5 ) ; @@ -182,13 +188,15 @@ long AuthAgent::CheckHttpResponse( const http::Header& hdr ) { // throw for other HTTP errors - if ( response >= 400 && response < 500 ) + if ( response >= 400 ) { - BOOST_THROW_EXCEPTION( - Error() - << HttpResponse( response ) - << Url( url ) - << HttpHeader( hdr ) ) ; + BOOST_THROW_EXCEPTION( + Error() + << HttpResponseCode( response ) + << HttpResponseHeaders( m_agent->LastErrorHeaders() ) + << HttpResponseText( m_agent->LastError() ) + << Url( url ) + << HttpRequestHeaders( hdr ) ) ; } return response ; diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index ec939a8d..797b2405 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -65,6 +65,9 @@ public : DataStream *dest, const http::Header& hdr ) ; + std::string LastError() const ; + std::string LastErrorHeaders() const ; + std::string RedirLocation() const ; std::string Escape( const std::string& str ) ; From 5da32905ea4be39cd85b60ed44e32d8492346150 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 27 Jun 2015 23:18:03 +0300 Subject: [PATCH 088/166] Move old tests to fix build errors, fixes #14 --- libgrive/CMakeLists.txt | 1 + libgrive/test/UnitTest.cc | 8 ++++---- libgrive/test/{drive => base}/ResourceTest.cc | 6 +++--- libgrive/test/{drive => base}/ResourceTest.hh | 0 libgrive/test/{drive => base}/ResourceTreeTest.cc | 5 ++--- libgrive/test/{drive => base}/ResourceTreeTest.hh | 0 libgrive/test/{drive => base}/StateTest.cc | 3 +-- libgrive/test/{drive => base}/StateTest.hh | 0 libgrive/test/drive/EntryTest.cc | 10 +++++----- libgrive/test/drive/EntryTest.hh | 6 +++--- 10 files changed, 19 insertions(+), 20 deletions(-) rename libgrive/test/{drive => base}/ResourceTest.cc (95%) rename libgrive/test/{drive => base}/ResourceTest.hh (100%) rename libgrive/test/{drive => base}/ResourceTreeTest.cc (91%) rename libgrive/test/{drive => base}/ResourceTreeTest.hh (100%) rename libgrive/test/{drive => base}/StateTest.cc (95%) rename libgrive/test/{drive => base}/StateTest.hh (100%) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 7fadd8dc..cbbde554 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -117,6 +117,7 @@ IF ( CPPUNIT_FOUND ) # list of test source files here file(GLOB TEST_SRC + test/base/*.cc test/drive/*.cc test/util/*.cc test/xml/*.cc diff --git a/libgrive/test/UnitTest.cc b/libgrive/test/UnitTest.cc index 19dcf503..c229ae60 100644 --- a/libgrive/test/UnitTest.cc +++ b/libgrive/test/UnitTest.cc @@ -22,9 +22,9 @@ #include "util/log/DefaultLog.hh" #include "drive/EntryTest.hh" -#include "drive/ResourceTest.hh" -#include "drive/ResourceTreeTest.hh" -#include "drive/StateTest.hh" +#include "base/ResourceTest.hh" +#include "base/ResourceTreeTest.hh" +#include "base/StateTest.hh" #include "util/DateTimeTest.hh" #include "util/FunctionTest.hh" #include "util/ConfigTest.hh" @@ -38,7 +38,7 @@ int main( int argc, char **argv ) gr::LogBase::Inst( std::auto_ptr(new gr::log::DefaultLog) ) ; CppUnit::TextUi::TestRunner runner; - runner.addTest( EntryTest::suite( ) ) ; + runner.addTest( Entry1Test::suite( ) ) ; runner.addTest( StateTest::suite( ) ) ; runner.addTest( ResourceTest::suite( ) ) ; runner.addTest( ResourceTreeTest::suite( ) ) ; diff --git a/libgrive/test/drive/ResourceTest.cc b/libgrive/test/base/ResourceTest.cc similarity index 95% rename from libgrive/test/drive/ResourceTest.cc rename to libgrive/test/base/ResourceTest.cc index 5f69c427..f6819034 100644 --- a/libgrive/test/drive/ResourceTest.cc +++ b/libgrive/test/base/ResourceTest.cc @@ -21,9 +21,9 @@ #include "Assert.hh" -#include "drive/Resource.hh" +#include "base/Resource.hh" -#include "drive/Entry.hh" +#include "drive/Entry1.hh" #include "xml/Node.hh" #include @@ -60,7 +60,7 @@ void ResourceTest::TestNormal( ) xml::Node entry = xml::Node::Element( "entry" ) ; entry.AddElement( "updated" ).AddText( "2012-05-09T16:13:22.401Z" ) ; - Entry remote( entry ) ; + Entry1 remote( entry ) ; subject.FromRemote( remote, DateTime() ) ; GRUT_ASSERT_EQUAL( "local_changed", subject.StateStr() ) ; } diff --git a/libgrive/test/drive/ResourceTest.hh b/libgrive/test/base/ResourceTest.hh similarity index 100% rename from libgrive/test/drive/ResourceTest.hh rename to libgrive/test/base/ResourceTest.hh diff --git a/libgrive/test/drive/ResourceTreeTest.cc b/libgrive/test/base/ResourceTreeTest.cc similarity index 91% rename from libgrive/test/drive/ResourceTreeTest.cc rename to libgrive/test/base/ResourceTreeTest.cc index 22285ade..c15df46a 100644 --- a/libgrive/test/drive/ResourceTreeTest.cc +++ b/libgrive/test/base/ResourceTreeTest.cc @@ -21,15 +21,14 @@ #include "Assert.hh" -#include "drive/ResourceTree.hh" -#include "drive/Resource.hh" +#include "base/ResourceTree.hh" +#include "base/Resource.hh" #include namespace grut { using namespace gr ; -using namespace gr::v1 ; ResourceTreeTest::ResourceTreeTest( ) { diff --git a/libgrive/test/drive/ResourceTreeTest.hh b/libgrive/test/base/ResourceTreeTest.hh similarity index 100% rename from libgrive/test/drive/ResourceTreeTest.hh rename to libgrive/test/base/ResourceTreeTest.hh diff --git a/libgrive/test/drive/StateTest.cc b/libgrive/test/base/StateTest.cc similarity index 95% rename from libgrive/test/drive/StateTest.cc rename to libgrive/test/base/StateTest.cc index 167dab27..6e7aff57 100644 --- a/libgrive/test/drive/StateTest.cc +++ b/libgrive/test/base/StateTest.cc @@ -21,7 +21,7 @@ #include "Assert.hh" -#include "drive/State.hh" +#include "base/State.hh" #include "json/Val.hh" #include "util/log/Log.hh" @@ -30,7 +30,6 @@ namespace grut { using namespace gr ; -using namespace gr::v1 ; StateTest::StateTest( ) { diff --git a/libgrive/test/drive/StateTest.hh b/libgrive/test/base/StateTest.hh similarity index 100% rename from libgrive/test/drive/StateTest.hh rename to libgrive/test/base/StateTest.hh diff --git a/libgrive/test/drive/EntryTest.cc b/libgrive/test/drive/EntryTest.cc index 48dfa5c7..a78516db 100644 --- a/libgrive/test/drive/EntryTest.cc +++ b/libgrive/test/drive/EntryTest.cc @@ -21,7 +21,7 @@ #include "Assert.hh" -#include "drive/Entry.hh" +#include "drive/Entry1.hh" #include "xml/Node.hh" #include "xml/NodeSet.hh" #include "xml/TreeBuilder.hh" @@ -33,17 +33,17 @@ namespace grut { using namespace gr ; using namespace gr::v1 ; -EntryTest::EntryTest( ) +Entry1Test::Entry1Test( ) { } -void EntryTest::TestXml( ) +void Entry1Test::TestXml( ) { xml::Node root = xml::TreeBuilder::ParseFile( TEST_DATA "entry.xml" ) ; CPPUNIT_ASSERT( !root["entry"].empty() ) ; - Entry subject( root["entry"].front() ) ; + Entry1 subject( root["entry"].front() ) ; GRUT_ASSERT_EQUAL( "snes", subject.Title() ) ; GRUT_ASSERT_EQUAL( "\"WxYPGE8CDyt7ImBk\"", subject.ETag() ) ; GRUT_ASSERT_EQUAL( "https://docs.google.com/feeds/default/private/full/folder%3A0B5KhdsbryVeGMl83OEV1ZVc3cUE", @@ -53,7 +53,7 @@ void EntryTest::TestXml( ) GRUT_ASSERT_EQUAL( "https://docs.google.com/feeds/default/private/full/folder%3A0B5KhdsbryVeGNEZjdUxzZHl3Sjg", subject.ParentHrefs().front() ) ; - GRUT_ASSERT_EQUAL( "folder", subject.Kind() ) ; + GRUT_ASSERT_EQUAL( true, subject.IsDir() ) ; } } // end of namespace grut diff --git a/libgrive/test/drive/EntryTest.hh b/libgrive/test/drive/EntryTest.hh index 8d4f7f47..f7bc8f51 100644 --- a/libgrive/test/drive/EntryTest.hh +++ b/libgrive/test/drive/EntryTest.hh @@ -24,13 +24,13 @@ namespace grut { -class EntryTest : public CppUnit::TestFixture +class Entry1Test : public CppUnit::TestFixture { public : - EntryTest( ) ; + Entry1Test( ) ; // declare suit function - CPPUNIT_TEST_SUITE( EntryTest ) ; + CPPUNIT_TEST_SUITE( Entry1Test ) ; CPPUNIT_TEST( TestXml ) ; CPPUNIT_TEST_SUITE_END(); From 1d476165187bcd179defe8a2b20ea5b85f03c17b Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 4 Jul 2015 00:18:59 +0300 Subject: [PATCH 089/166] Fix #17 (curl_slist memory leak) --- libgrive/src/http/CurlAgent.cc | 11 ++++++++--- libgrive/src/http/CurlAgent.hh | 1 - 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index bc766b77..04c11434 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -91,6 +91,8 @@ struct CurlAgent::Impl DataStream *dest ; } ; +static struct curl_slist* SetHeader( CURL* handle, const Header& hdr ); + CurlAgent::CurlAgent() : m_pimpl( new Impl ) { @@ -166,10 +168,12 @@ long CurlAgent::ExecCurl( ::curl_easy_setopt(curl, CURLOPT_WRITEDATA, this ) ; m_pimpl->dest = dest ; - SetHeader( hdr ) ; + struct curl_slist *slist = SetHeader( m_pimpl->curl, hdr ) ; CURLcode curl_code = ::curl_easy_perform(curl); + curl_slist_free_all(slist); + // get the HTTP response code long http_code = 0; ::curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code); @@ -288,14 +292,15 @@ long CurlAgent::Custom( return ExecCurl( url, dest, hdr ) ; } -void CurlAgent::SetHeader( const Header& hdr ) +static struct curl_slist* SetHeader( CURL *handle, const Header& hdr ) { // set headers struct curl_slist *curl_hdr = 0 ; for ( Header::iterator i = hdr.begin() ; i != hdr.end() ; ++i ) curl_hdr = curl_slist_append( curl_hdr, i->c_str() ) ; - ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HTTPHEADER, curl_hdr ) ; + ::curl_easy_setopt( handle, CURLOPT_HTTPHEADER, curl_hdr ) ; + return curl_hdr; } std::string CurlAgent::LastError() const diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index b7125336..e06fd212 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -82,7 +82,6 @@ private : static std::size_t HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; static std::size_t Receive( void* ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; - void SetHeader( const Header& hdr ) ; long ExecCurl( const std::string& url, DataStream *dest, From 308aab92cfdbee11dbd608a30b2fa77ddade1b31 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 4 Jul 2015 00:25:13 +0300 Subject: [PATCH 090/166] Also print response body on 401 --- libgrive/src/protocol/AuthAgent.cc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index e24d4e02..b7cb0c9c 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -161,7 +161,7 @@ bool AuthAgent::CheckRetry( long response ) // HTTP 500 and 503 should be temporary. just wait a bit and retry if ( response == 500 || response == 503 ) { - Log( "request failed due to temporary error: %1%, body: %2%. retrying in 5 seconds", + Log( "request failed due to temporary error: %1% (body: %2%). retrying in 5 seconds", response, m_agent->LastError(), log::warning ) ; os::Sleep( 5 ) ; @@ -171,8 +171,8 @@ bool AuthAgent::CheckRetry( long response ) // HTTP 401 Unauthorized. the auth token has been expired. refresh it else if ( response == 401 ) { - Log( "request failed due to auth token expired: %1%. refreshing token", - response, log::warning ) ; + Log( "request failed due to auth token expired: %1% (body: %2%). refreshing token", + response, m_agent->LastError(), log::warning ) ; os::Sleep( 5 ) ; m_auth.Refresh() ; From 6f35a32ca2a61af5bfef02b5ab10583d567b37c3 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Tue, 4 Aug 2015 22:59:09 +0300 Subject: [PATCH 091/166] Get auth headers again before each Put() retry - fixes #19 --- libgrive/src/protocol/AuthAgent.cc | 79 ++++++++++++++++-------------- 1 file changed, 41 insertions(+), 38 deletions(-) diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index b7cb0c9c..6af5be2d 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -52,13 +52,14 @@ long AuthAgent::Put( DataStream *dest, const Header& hdr ) { - Header auth = AppendHeader(hdr) ; - - long response ; - while ( CheckRetry( - response = m_agent->Put(url, data, dest, auth) ) ) ; - - return CheckHttpResponse(response, url, auth) ; + long response; + Header auth; + do + { + auth = AppendHeader( hdr ); + response = m_agent->Put( url, data, dest, auth ); + } while ( CheckRetry( response ) ); + return CheckHttpResponse( response, url, auth ); } long AuthAgent::Put( @@ -67,16 +68,15 @@ long AuthAgent::Put( DataStream *dest, const Header& hdr ) { - Header auth = AppendHeader(hdr) ; - - long response ; - bool keepTrying = true; - while ( keepTrying ) { + long response; + Header auth; + while ( true ) + { + auth = AppendHeader( hdr ); response = m_agent->Put( url, file, dest, auth ); - keepTrying = CheckRetry( response ); - if ( keepTrying ) { - file->Seek( 0, SEEK_SET ); - } + if ( !CheckRetry( response ) ) + break; + file->Seek( 0, SEEK_SET ); } // On 410 Gone or 412 Precondition failed, recovery may be possible so don't @@ -84,7 +84,7 @@ long AuthAgent::Put( if ( response == 410 || response == 412 ) return response; - return CheckHttpResponse(response, url, auth) ; + return CheckHttpResponse( response, url, auth ); } long AuthAgent::Get( @@ -92,13 +92,14 @@ long AuthAgent::Get( DataStream *dest, const Header& hdr ) { - Header auth = AppendHeader(hdr) ; - - long response ; - while ( CheckRetry( - response = m_agent->Get( url, dest, AppendHeader(hdr) ) ) ) ; - - return CheckHttpResponse(response, url, auth) ; + long response; + Header auth; + do + { + auth = AppendHeader( hdr ); + response = m_agent->Get( url, dest, auth ); + } while ( CheckRetry( response ) ); + return CheckHttpResponse( response, url, auth ); } long AuthAgent::Post( @@ -107,13 +108,14 @@ long AuthAgent::Post( DataStream *dest, const Header& hdr ) { - Header auth = AppendHeader(hdr) ; - - long response ; - while ( CheckRetry( - response = m_agent->Post( url, data, dest, AppendHeader(hdr) ) ) ) ; - - return CheckHttpResponse(response, url, auth) ; + long response; + Header auth; + do + { + auth = AppendHeader( hdr ); + response = m_agent->Post( url, data, dest, auth ); + } while ( CheckRetry( response ) ); + return CheckHttpResponse( response, url, auth ); } long AuthAgent::Custom( @@ -122,13 +124,14 @@ long AuthAgent::Custom( DataStream *dest, const Header& hdr ) { - Header auth = AppendHeader(hdr) ; - - long response ; - while ( CheckRetry( - response = m_agent->Custom( method, url, dest, AppendHeader(hdr) ) ) ) ; - - return CheckHttpResponse(response, url, auth) ; + long response; + Header auth; + do + { + auth = AppendHeader( hdr ); + response = m_agent->Custom( method, url, dest, auth ); + } while ( CheckRetry( response ) ); + return CheckHttpResponse( response, url, auth ); } std::string AuthAgent::LastError() const From 5dc7028c8aebb26d35ed31c4b07ac180cc1887c9 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 30 Aug 2015 22:48:30 +0300 Subject: [PATCH 092/166] Add expat into README.md dependency list - closes #25 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 146828c7..433fa4dc 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ You need the following libraries: - libstdc++ - libgcrypt - Boost (Boost filesystem and program_option are required) +- expat There are also some optional dependencies: - CppUnit (for unit tests) From 7ac15187d3af3b05b85e6c98b22f6a47155993c3 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 25 Sep 2015 13:01:26 +0300 Subject: [PATCH 093/166] Add --ignore (perl regexp) option --- grive/src/main.cc | 1 + libgrive/CMakeLists.txt | 2 +- libgrive/src/base/Resource.cc | 9 +++++++++ libgrive/src/base/Resource.hh | 1 + libgrive/src/base/State.cc | 33 +++++++++++++++++++++------------ libgrive/src/base/State.hh | 4 +++- libgrive/src/util/Config.cc | 3 +++ 7 files changed, 39 insertions(+), 14 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index f151e4b9..07fd6132 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -120,6 +120,7 @@ int Main( int argc, char **argv ) "instead of uploading it." ) ( "dry-run", "Only detect which files need to be uploaded/downloaded, " "without actually performing them." ) + ( "ignore", po::value(), "Ignore files relative paths of which match this Perl RegExp." ) ; po::variables_map vm; diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index cbbde554..f3aa7757 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -5,7 +5,7 @@ set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/") find_package(LibGcrypt REQUIRED) find_package(CURL REQUIRED) find_package(EXPAT REQUIRED) -find_package(Boost 1.40.0 COMPONENTS program_options filesystem unit_test_framework system REQUIRED) +find_package(Boost 1.40.0 COMPONENTS program_options filesystem unit_test_framework regex system REQUIRED) find_package(BFD) find_package(CppUnit) find_package(Iberty) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 058f08f8..e206ebc5 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -337,6 +337,15 @@ fs::path Resource::Path() const return m_parent != 0 ? (m_parent->Path() / m_name) : m_name ; } +// Path relative to the root directory +fs::path Resource::RelPath() const +{ + assert( m_parent != this ) ; + assert( m_parent == 0 || m_parent->IsFolder() ) ; + + return m_parent != 0 && !m_parent->IsRoot() ? (m_parent->Path() / m_name) : m_name ; +} + bool Resource::IsInRootTree() const { assert( m_parent == 0 || m_parent->IsFolder() ) ; diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index efb0d5f1..06a61a4f 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -100,6 +100,7 @@ public : Resource* FindChild( const std::string& title ) ; fs::path Path() const ; + fs::path RelPath() const ; bool IsInRootTree() const ; bool IsRoot() const ; bool HasID() const ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 4aaaf220..fa1e037d 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -36,11 +36,12 @@ namespace gr { State::State( const fs::path& filename, const Val& options ) : m_res ( options["path"].Str() ), m_dir ( options["dir"].Str() ), - m_cstamp ( -1 ) + m_cstamp ( -1 ), + m_ign ( !options["ignore"].Str().empty() ? options["ignore"].Str()+"|^\\.(grive|grive_state|trash)" : "^\\.(grive|grive_state|trash)" ) { Read( filename ) ; - // the "-f" option will make grive always thinks remote is newer + // the "-f" option will make grive always think remote is newer Val force ; if ( options.Get("force", force) && force.Bool() ) m_last_sync = DateTime() ; @@ -61,7 +62,7 @@ void State::FromLocal( const fs::path& p ) bool State::IsIgnore( const std::string& filename ) { - return filename == ".grive" || filename == ".grive_state" || filename == ".trash"; + return regex_match( filename.c_str(), m_ign ); } void State::FromLocal( const fs::path& p, Resource* folder ) @@ -77,8 +78,9 @@ void State::FromLocal( const fs::path& p, Resource* folder ) std::string fname = i->path().filename().string() ; fs::file_status st = fs::status(i->path()); - if ( IsIgnore(fname) ) - Log( "file %1% is ignored by grive", fname, log::verbose ) ; + std::string path = folder->IsRoot() ? fname : ( folder->RelPath() / fname ).string(); + if ( IsIgnore( path ) ) + Log( "file %1% is ignored by grive", path, log::verbose ) ; // check if it is ignored else if ( folder == m_res.Root() && m_dir != "" && fname != m_dir ) @@ -114,11 +116,7 @@ void State::FromRemote( const Entry& e ) std::string fn = e.Filename() ; std::string k = e.IsDir() ? "folder" : "file"; - if ( IsIgnore( e.Name() ) ) - Log( "%1% %2% is ignored by grive", k, e.Name(), log::verbose ) ; - - // check if it is ignored - else if ( e.ParentHref() == m_res.Root()->SelfHref() && m_dir != "" && e.Name() != m_dir ) + if ( e.ParentHref() == m_res.Root()->SelfHref() && m_dir != "" && e.Name() != m_dir ) Log( "%1% %2% is ignored", k, e.Name(), log::verbose ); // common checkings @@ -170,7 +168,6 @@ std::size_t State::TryResolveEntry() void State::FromChange( const Entry& e ) { assert( e.IsChange() ) ; - assert( !IsIgnore( e.Name() ) ) ; // entries in the change feed is always treated as newer in remote, // so we override the last sync time to 0 @@ -185,13 +182,25 @@ bool State::Update( const Entry& e ) if ( Resource *res = m_res.FindByHref( e.SelfHref() ) ) { + std::string path = res->RelPath().string(); + if ( IsIgnore( path ) ) + { + Log( "%1% is ignored by grive", path, log::verbose ) ; + return true; + } m_res.Update( res, e, m_last_sync ) ; - return true ; } else if ( Resource *parent = m_res.FindByHref( e.ParentHref() ) ) { assert( parent->IsFolder() ) ; + std::string path = parent->IsRoot() ? e.Name() : ( parent->RelPath() / e.Name() ).string(); + if ( IsIgnore( path ) ) + { + Log( "%1% is ignored by grive", path, log::verbose ) ; + return true; + } + // see if the entry already exist in local std::string name = e.Name() ; Resource *child = parent->FindChild( name ) ; diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index c870cae2..acbb6994 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -25,6 +25,7 @@ #include "util/FileSystem.hh" #include +#include namespace gr { @@ -69,13 +70,14 @@ private : bool Update( const Entry& e ) ; std::size_t TryResolveEntry() ; - static bool IsIgnore( const std::string& filename ) ; + bool IsIgnore( const std::string& filename ) ; private : ResourceTree m_res ; DateTime m_last_sync ; int m_cstamp ; std::string m_dir ; + boost::regex m_ign ; std::vector m_unresolved ; } ; diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index ca7c86ad..c1c53ea0 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -47,6 +47,9 @@ Config::Config( const po::variables_map& vm ) m_cmd.Add( "dir", Val(vm.count("dir") > 0 ? vm["dir"].as() : "" ) ) ; + m_cmd.Add( "ignore", Val(vm.count("ignore") > 0 + ? vm["ignore"].as() + : "" ) ) ; m_path = GetPath( fs::path(m_cmd["path"].Str()) ) ; m_file = Read( ) ; From 8fd74d7a36ebd0d527347834ef9dc1c1fd51a546 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 25 Sep 2015 13:06:40 +0300 Subject: [PATCH 094/166] Update readme --- README.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 433fa4dc..676efab9 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# Grive2 0.4.0 +# Grive2 0.4.1 -17 May 2015, Vitaliy Filippov +4 Aug 2015, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 @@ -28,7 +28,7 @@ You need the following libraries: - libcurl - libstdc++ - libgcrypt -- Boost (Boost filesystem and program_option are required) +- Boost (Boost filesystem, program_options and regex are required) - expat There are also some optional dependencies: @@ -60,7 +60,15 @@ Enjoy! ## Version History -### Grive2 v0.4 +### Grive2 v0.4.2 (unreleased) + +- Exclude files by perl regexp + +### Grive2 v0.4.1 + +- Bug fixes + +### Grive2 v0.4.0 First fork release, by Vitaliy Filippov / vitalif at mail*ru - Support for the new Google Drive REST API (old "Document List" API is shut down by Google 20 April 2015) From 9b8e0c826bde7b8fc359ba81e58ed2b448c00a46 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 30 Sep 2015 13:53:53 +0300 Subject: [PATCH 095/166] Remove request logging from Drive/Feed/Syncer (prepare to move it into CurlAgent) --- CMakeLists.txt | 2 +- libgrive/src/base/Drive.cc | 5 ----- libgrive/src/base/Feed.cc | 8 -------- libgrive/src/base/Feed.hh | 8 -------- libgrive/src/drive/Feed1.cc | 10 +--------- libgrive/src/drive/Syncer1.cc | 2 -- libgrive/src/drive2/Syncer2.cc | 1 - 7 files changed, 2 insertions(+), 34 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 293a08bd..c810ff9f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! -set( GRIVE_VERSION "0.4.0-pre" ) +set( GRIVE_VERSION "0.4.2-pre" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index 10a194b4..a8efb8a4 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -129,9 +129,6 @@ void Drive::DetectChanges() Log( "Reading remote server file list", log::info ) ; std::auto_ptr feed = m_syncer->GetAll() ; - if ( m_options["log-xml"].Bool() ) - feed->EnableLog( "/tmp/file", ".xml" ) ; - while ( feed->GetNext( m_syncer->Agent() ) ) { std::for_each( @@ -144,8 +141,6 @@ void Drive::DetectChanges() { Log( "Detecting changes from last sync", log::info ) ; feed = m_syncer->GetChanges( prev_stamp+1 ) ; - if ( m_options["log-xml"].Bool() ) - feed->EnableLog( "/tmp/changes", ".xml" ) ; while ( feed->GetNext( m_syncer->Agent() ) ) { std::for_each( diff --git a/libgrive/src/base/Feed.cc b/libgrive/src/base/Feed.cc index 7ff38508..0ea834d3 100644 --- a/libgrive/src/base/Feed.cc +++ b/libgrive/src/base/Feed.cc @@ -40,12 +40,4 @@ Feed::iterator Feed::end() const return m_entries.end() ; } -void Feed::EnableLog( const std::string& prefix, const std::string& suffix ) -{ - m_log.reset( new LogInfo ) ; - m_log->prefix = prefix ; - m_log->suffix = suffix ; - m_log->sequence = 0 ; -} - } // end of namespace gr::v1 diff --git a/libgrive/src/base/Feed.hh b/libgrive/src/base/Feed.hh index 8be1e55f..d43c68ce 100644 --- a/libgrive/src/base/Feed.hh +++ b/libgrive/src/base/Feed.hh @@ -43,16 +43,8 @@ public : virtual bool GetNext( http::Agent *http ) = 0 ; iterator begin() const ; iterator end() const ; - virtual void EnableLog( const std::string& prefix, const std::string& suffix ) ; protected : - struct LogInfo - { - std::string prefix ; - std::string suffix ; - std::size_t sequence ; - } ; - std::auto_ptr m_log ; Entries m_entries ; std::string m_next ; } ; diff --git a/libgrive/src/drive/Feed1.cc b/libgrive/src/drive/Feed1.cc index addde4bb..86d59bd7 100644 --- a/libgrive/src/drive/Feed1.cc +++ b/libgrive/src/drive/Feed1.cc @@ -24,7 +24,6 @@ #include "http/Agent.hh" #include "http/Header.hh" -#include "http/ResponseLog.hh" #include "http/XmlResponse.hh" #include "xml/NodeSet.hh" @@ -42,18 +41,11 @@ Feed1::Feed1( const std::string &url ): bool Feed1::GetNext( http::Agent *http ) { http::XmlResponse xrsp ; - http::ResponseLog log( &xrsp ) ; if ( m_next.empty() ) return false; - if ( m_log.get() != 0 ) - log.Reset( - m_log->prefix, - (boost::format( "-#%1%%2%" ) % m_log->sequence++ % m_log->suffix ).str(), - &xrsp ) ; - - http->Get( m_next, &log, http::Header() ) ; + http->Get( m_next, &xrsp, http::Header() ) ; xml::Node m_root = xrsp.Response() ; xml::NodeSet xe = m_root["entry"] ; diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index 31a331a0..1374a5e4 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -25,7 +25,6 @@ #include "http/Agent.hh" #include "http/Header.hh" -//#include "http/ResponseLog.hh" #include "http/StringResponse.hh" #include "http/XmlResponse.hh" @@ -123,7 +122,6 @@ bool Syncer1::Create( Resource *res ) hdr.Add( "Content-Type: application/atom+xml" ) ; http::XmlResponse xml ; -// http::ResponseLog log( "create", ".xml", &xml ) ; m_http->Post( uri, meta, &xml, hdr ) ; AssignIDs( res, Entry1( xml.Response() ) ) ; diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index d0f6f3c7..eb5e943a 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -27,7 +27,6 @@ #include "http/Download.hh" #include "http/Header.hh" #include "http/StringResponse.hh" -//#include "http/ResponseLog.hh" #include "json/ValResponse.hh" #include "json/JsonWriter.hh" From 9402bff12e50621fcd7f70779b78b552a3d5dae6 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 30 Sep 2015 15:01:50 +0300 Subject: [PATCH 096/166] Implement normal reliable HTTP logging (in Agent) Flushes on every call so may be slightly slow, but in return always provides full information in case of a segfault or a failed assertion. --- grive/src/main.cc | 15 ++++++++---- libgrive/src/http/Agent.hh | 4 +++ libgrive/src/http/CurlAgent.cc | 16 ++++++++++++ libgrive/src/http/CurlAgent.hh | 6 ++++- libgrive/src/http/ResponseLog.cc | 39 +++++++++--------------------- libgrive/src/http/ResponseLog.hh | 12 +++------ libgrive/src/json/ValBuilder.cc | 13 ++++++---- libgrive/src/protocol/AuthAgent.cc | 12 ++++++++- libgrive/src/protocol/AuthAgent.hh | 7 ++++-- libgrive/src/protocol/OAuth2.cc | 32 +++++++++++------------- libgrive/src/protocol/OAuth2.hh | 19 +++++++++------ libgrive/src/util/Config.cc | 1 - 12 files changed, 99 insertions(+), 77 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index 07fd6132..a6efff5d 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -112,7 +112,7 @@ int Main( int argc, char **argv ) ( "path,p", po::value(), "Path to sync") ( "dir,s", po::value(), "Subdirectory to sync") ( "verbose,V", "Verbose mode. Enable more messages than normal.") - ( "log-xml", "Log more HTTP responses as XML for debugging.") + ( "log-http", po::value(), "Log all HTTP responses in this file for debugging.") ( "new-rev", "Create new revisions in server for updated files.") ( "debug,d", "Enable debug level messages. Implies -v.") ( "log,l", po::value(), "Set log output filename." ) @@ -147,12 +147,18 @@ int Main( int argc, char **argv ) Log( "config file name %1%", config.Filename(), log::verbose ); + std::auto_ptr http( new http::CurlAgent ); + if ( vm.count( "log-http" ) ) + http->SetLog( new http::ResponseLog( vm["log-http"].as(), ".txt" ) ); + if ( vm.count( "auth" ) ) { + OAuth2 token( http, client_id, client_secret ) ; + std::cout << "-----------------------\n" << "Please go to this URL and get an authentication code:\n\n" - << OAuth2::MakeAuthURL( client_id ) + << token.MakeAuthURL() << std::endl ; std::cout @@ -161,7 +167,6 @@ int Main( int argc, char **argv ) std::string code ; std::cin >> code ; - OAuth2 token( client_id, client_secret ) ; token.Auth( code ) ; // save to config @@ -184,8 +189,8 @@ int Main( int argc, char **argv ) return -1; } - OAuth2 token( refresh_token, client_id, client_secret ) ; - AuthAgent agent( token, std::auto_ptr( new http::CurlAgent ) ) ; + OAuth2 token( http, refresh_token, client_id, client_secret ) ; + AuthAgent agent( token, http ) ; v2::Syncer2 syncer( &agent ); Drive drive( &syncer, config.GetAll() ) ; diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index 5f4d5dec..09d3b87d 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -20,6 +20,7 @@ #pragma once #include +#include "ResponseLog.hh" namespace gr { @@ -35,6 +36,9 @@ class Agent public : virtual ~Agent() {} + virtual ResponseLog* GetLog() const = 0 ; + virtual void SetLog( ResponseLog* ) = 0 ; + virtual long Put( const std::string& url, const std::string& data, diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 04c11434..2e9d5668 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -118,6 +118,16 @@ CurlAgent::~CurlAgent() ::curl_easy_cleanup( m_pimpl->curl ); } +ResponseLog* CurlAgent::GetLog() const +{ + return m_log.get(); +} + +void CurlAgent::SetLog(ResponseLog *log) +{ + m_log.reset( log ); +} + std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) { char *str = static_cast(ptr) ; @@ -130,6 +140,9 @@ std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, Cur if ( pthis->m_pimpl->error ) pthis->m_pimpl->error_headers += line; + if ( pthis->m_log.get() ) + pthis->m_log->Write( str, size*nmemb ); + static const std::string loc = "Location: " ; std::size_t pos = line.find( loc ) ; if ( pos != line.npos ) @@ -144,6 +157,8 @@ std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, Cur std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, CurlAgent *pthis ) { assert( pthis != 0 ) ; + if ( pthis->m_log.get() ) + pthis->m_log->Write( (const char*)ptr, size*nmemb ); if ( pthis->m_pimpl->error && pthis->m_pimpl->error_data.size() < 65536 ) { // Do not feed error responses to destination stream @@ -284,6 +299,7 @@ long CurlAgent::Custom( { Trace("HTTP %2% \"%1%\"", url, method ) ; + Init() ; CURL *curl = m_pimpl->curl ; ::curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, method.c_str() ); diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index e06fd212..044e2bc0 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -40,7 +40,10 @@ class CurlAgent : public Agent public : CurlAgent() ; ~CurlAgent() ; - + + ResponseLog* GetLog() const ; + void SetLog( ResponseLog *log ) ; + long Put( const std::string& url, const std::string& post_data, @@ -92,6 +95,7 @@ private : private : struct Impl ; std::auto_ptr m_pimpl ; + std::auto_ptr m_log ; } ; } } // end of namespace diff --git a/libgrive/src/http/ResponseLog.cc b/libgrive/src/http/ResponseLog.cc index 9de4ac04..b26856e5 100644 --- a/libgrive/src/http/ResponseLog.cc +++ b/libgrive/src/http/ResponseLog.cc @@ -28,19 +28,9 @@ namespace gr { namespace http { ResponseLog::ResponseLog( const std::string& prefix, - const std::string& suffix, - DataStream *next ) : - m_enabled ( true ), - m_next ( next ) + const std::string& suffix ) { - Reset( prefix, suffix, next ) ; -} - -ResponseLog::ResponseLog( DataStream *next ) : - m_enabled ( false ), - m_next ( next ) -{ - assert( m_next != 0 ) ; + Reset( prefix, suffix ) ; } std::size_t ResponseLog::Write( const char *data, std::size_t count ) @@ -49,15 +39,14 @@ std::size_t ResponseLog::Write( const char *data, std::size_t count ) { assert( m_log.rdbuf() != 0 ) ; m_log.rdbuf()->sputn( data, count ) ; + m_log.flush(); } - - return m_next->Write( data, count ) ; + return count; } std::size_t ResponseLog::Read( char *data, std::size_t count ) { - assert( m_next != 0 ) ; - return m_next->Read( data, count ) ; + return 0 ; } std::string ResponseLog::Filename( const std::string& prefix, const std::string& suffix ) @@ -65,19 +54,12 @@ std::string ResponseLog::Filename( const std::string& prefix, const std::string& return prefix + DateTime::Now().Format( "%F.%H%M%S" ) + suffix ; } -void ResponseLog::Enable( bool enable ) +void ResponseLog::Reset( const std::string& prefix, const std::string& suffix ) { - m_enabled = enable ; -} - -void ResponseLog::Reset( const std::string& prefix, const std::string& suffix, DataStream *next ) -{ - assert( next != 0 ) ; - if ( m_log.is_open() ) m_log.close() ; - const std::string fname = Filename(prefix, suffix) ; + const std::string fname = Filename( prefix, suffix ) ; // reset previous stream state. don't care if file can be opened // successfully previously @@ -88,12 +70,13 @@ void ResponseLog::Reset( const std::string& prefix, const std::string& suffix, D if ( m_log ) { Trace( "logging HTTP response: %1%", fname ) ; - m_enabled = true ; + m_enabled = true ; } else + { Trace( "cannot open log file %1%", fname ) ; - - m_next = next ; + m_enabled = false ; + } } }} // end of namespace diff --git a/libgrive/src/http/ResponseLog.hh b/libgrive/src/http/ResponseLog.hh index 046b5c97..9c3b63a9 100644 --- a/libgrive/src/http/ResponseLog.hh +++ b/libgrive/src/http/ResponseLog.hh @@ -31,23 +31,19 @@ class ResponseLog : public DataStream public : ResponseLog( const std::string& prefix, - const std::string& suffix, - DataStream *next ) ; - ResponseLog( DataStream *next ) ; - + const std::string& suffix ) ; + std::size_t Write( const char *data, std::size_t count ) ; std::size_t Read( char *data, std::size_t count ) ; - void Enable( bool enable = true ) ; - void Reset( const std::string& prefix, const std::string& suffix, DataStream *next ) ; + void Reset( const std::string& prefix, const std::string& suffix ) ; private : static std::string Filename( const std::string& prefix, const std::string& suffix ) ; private : - bool m_enabled ; + bool m_enabled ; std::ofstream m_log ; - DataStream *m_next ; } ; } } // end of namespace diff --git a/libgrive/src/json/ValBuilder.cc b/libgrive/src/json/ValBuilder.cc index c5658ff2..7a0068ec 100644 --- a/libgrive/src/json/ValBuilder.cc +++ b/libgrive/src/json/ValBuilder.cc @@ -20,8 +20,6 @@ #include "ValBuilder.hh" -#include - namespace gr { ValBuilder::ValBuilder( ) @@ -106,7 +104,8 @@ void ValBuilder::End( Val::TypeEnum type ) { if ( m_ctx.top().val.Type() == type ) { - assert( m_ctx.top().key.Is() ) ; + if( !m_ctx.top().key.Is() ) + BOOST_THROW_EXCEPTION( Error() << Unexpected_(m_ctx.top().key) ) ; // get top Val from stack Val current ; @@ -130,8 +129,12 @@ void ValBuilder::EndObject() Val ValBuilder::Result() const { - assert( m_ctx.size() == 1U ) ; - return m_ctx.top().val ; + if ( !m_ctx.size() ) + BOOST_THROW_EXCEPTION( Error() << NoKey_( Val(std::string("")) ) ) ; + Val r = m_ctx.top().val; + if ( m_ctx.size() > 0 ) + BOOST_THROW_EXCEPTION( Error() << Unexpected_(m_ctx.top().val) ) ; + return r; } } // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 6af5be2d..145c05d4 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -31,13 +31,23 @@ namespace gr { using namespace http ; -AuthAgent::AuthAgent( const OAuth2& auth, std::auto_ptr real_agent ) : +AuthAgent::AuthAgent( OAuth2& auth, std::auto_ptr real_agent ) : m_auth ( auth ), m_agent ( real_agent ) { assert( m_agent.get() != 0 ) ; } +http::ResponseLog* AuthAgent::GetLog() const +{ + return m_agent->GetLog(); +} + +void AuthAgent::SetLog( http::ResponseLog *log ) +{ + return m_agent->SetLog( log ); +} + Header AuthAgent::AppendHeader( const Header& hdr ) const { Header h(hdr) ; diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index 797b2405..4ceeaffc 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -34,7 +34,10 @@ namespace gr { class AuthAgent : public http::Agent { public : - AuthAgent( const OAuth2& auth, std::auto_ptr real_agent ) ; + AuthAgent( OAuth2& auth, std::auto_ptr real_agent ) ; + + http::ResponseLog* GetLog() const ; + void SetLog( http::ResponseLog *log ) ; long Put( const std::string& url, @@ -82,7 +85,7 @@ private : const http::Header& hdr ) ; private : - OAuth2 m_auth ; + OAuth2& m_auth ; const std::auto_ptr m_agent ; } ; diff --git a/libgrive/src/protocol/OAuth2.cc b/libgrive/src/protocol/OAuth2.cc index a13c8e3e..6dbd754b 100644 --- a/libgrive/src/protocol/OAuth2.cc +++ b/libgrive/src/protocol/OAuth2.cc @@ -33,9 +33,11 @@ namespace gr { const std::string token_url = "https://accounts.google.com/o/oauth2/token" ; OAuth2::OAuth2( + std::auto_ptr& agent, const std::string& refresh_code, const std::string& client_id, const std::string& client_secret ) : + m_agent( agent ), m_refresh( refresh_code ), m_client_id( client_id ), m_client_secret( client_secret ) @@ -44,8 +46,10 @@ OAuth2::OAuth2( } OAuth2::OAuth2( + std::auto_ptr& agent, const std::string& client_id, const std::string& client_secret ) : + m_agent( agent ), m_client_id( client_id ), m_client_secret( client_secret ) { @@ -61,32 +65,26 @@ void OAuth2::Auth( const std::string& auth_code ) "&grant_type=authorization_code" ; http::ValResponse resp ; - http::CurlAgent http ; - DisableLog dlog( log::debug ) ; - http.Post( token_url, post, &resp, http::Header() ) ; + m_agent->Post( token_url, post, &resp, http::Header() ) ; Val jresp = resp.Response() ; m_access = jresp["access_token"].Str() ; m_refresh = jresp["refresh_token"].Str() ; } -std::string OAuth2::MakeAuthURL( - const std::string& client_id, - const std::string& state ) +std::string OAuth2::MakeAuthURL() { - http::CurlAgent h ; - return "https://accounts.google.com/o/oauth2/auth" "?scope=" + - h.Escape( "https://www.googleapis.com/auth/userinfo.email" ) + "+" + - h.Escape( "https://www.googleapis.com/auth/userinfo.profile" ) + "+" + - h.Escape( "https://docs.google.com/feeds/" ) + "+" + - h.Escape( "https://docs.googleusercontent.com/" ) + "+" + - h.Escape( "https://spreadsheets.google.com/feeds/" ) + + m_agent->Escape( "https://www.googleapis.com/auth/userinfo.email" ) + "+" + + m_agent->Escape( "https://www.googleapis.com/auth/userinfo.profile" ) + "+" + + m_agent->Escape( "https://docs.google.com/feeds/" ) + "+" + + m_agent->Escape( "https://docs.googleusercontent.com/" ) + "+" + + m_agent->Escape( "https://spreadsheets.google.com/feeds/" ) + "&redirect_uri=urn:ietf:wg:oauth:2.0:oob" "&response_type=code" - "&client_id=" + client_id ; + "&client_id=" + m_client_id ; } void OAuth2::Refresh( ) @@ -98,10 +96,8 @@ void OAuth2::Refresh( ) "&grant_type=refresh_token" ; http::ValResponse resp ; - http::CurlAgent http ; - - DisableLog dlog( log::debug ) ; - http.Post( token_url, post, &resp, http::Header() ) ; + + m_agent->Post( token_url, post, &resp, http::Header() ) ; m_access = resp.Response()["access_token"].Str() ; } diff --git a/libgrive/src/protocol/OAuth2.hh b/libgrive/src/protocol/OAuth2.hh index e8713f3e..71c1c014 100644 --- a/libgrive/src/protocol/OAuth2.hh +++ b/libgrive/src/protocol/OAuth2.hh @@ -19,7 +19,9 @@ #pragma once +#include "http/Agent.hh" #include +#include namespace gr { @@ -27,34 +29,35 @@ class OAuth2 { public : OAuth2( + std::auto_ptr& agent, const std::string& client_id, const std::string& client_secret ) ; OAuth2( + std::auto_ptr& agent, const std::string& refresh_code, const std::string& client_id, const std::string& client_secret ) ; std::string Str() const ; - - static std::string MakeAuthURL( - const std::string& client_id, - const std::string& state = std::string() ) ; + + std::string MakeAuthURL() ; void Auth( const std::string& auth_code ) ; void Refresh( ) ; - + std::string RefreshToken( ) const ; std::string AccessToken( ) const ; - + // adding HTTP auth header std::string HttpHeader( ) const ; - + private : std::string m_access ; std::string m_refresh ; + std::auto_ptr m_agent ; const std::string m_client_id ; const std::string m_client_secret ; } ; - + } // end of namespace diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index c1c53ea0..69ac344c 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -38,7 +38,6 @@ const std::string default_root_folder = "."; Config::Config( const po::variables_map& vm ) { - m_cmd.Add( "log-xml", Val(vm.count("log-xml") > 0) ) ; m_cmd.Add( "new-rev", Val(vm.count("new-rev") > 0) ) ; m_cmd.Add( "force", Val(vm.count("force") > 0 ) ) ; m_cmd.Add( "path", Val(vm.count("path") > 0 From dc8e172a27efb7e208de57cf36d20a10f650386c Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 30 Sep 2015 15:52:24 +0300 Subject: [PATCH 097/166] Handle error HTTP responses in OAuth2 --- libgrive/src/protocol/OAuth2.cc | 29 ++++++++++++++++++++++------- libgrive/src/protocol/OAuth2.hh | 4 ++++ 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/libgrive/src/protocol/OAuth2.cc b/libgrive/src/protocol/OAuth2.cc index 6dbd754b..be355160 100644 --- a/libgrive/src/protocol/OAuth2.cc +++ b/libgrive/src/protocol/OAuth2.cc @@ -66,11 +66,19 @@ void OAuth2::Auth( const std::string& auth_code ) http::ValResponse resp ; - m_agent->Post( token_url, post, &resp, http::Header() ) ; - - Val jresp = resp.Response() ; - m_access = jresp["access_token"].Str() ; - m_refresh = jresp["refresh_token"].Str() ; + long code = m_agent->Post( token_url, post, &resp, http::Header() ) ; + if ( code >= 200 && code < 300 ) + { + Val jresp = resp.Response() ; + m_access = jresp["access_token"].Str() ; + m_refresh = jresp["refresh_token"].Str() ; + } + else + { + Log( "Failed to obtain auth token: HTTP %1%, body: %2%", + code, m_agent->LastError(), log::error ) ; + BOOST_THROW_EXCEPTION( AuthFailed() ); + } } std::string OAuth2::MakeAuthURL() @@ -97,9 +105,16 @@ void OAuth2::Refresh( ) http::ValResponse resp ; - m_agent->Post( token_url, post, &resp, http::Header() ) ; + long code = m_agent->Post( token_url, post, &resp, http::Header() ) ; - m_access = resp.Response()["access_token"].Str() ; + if ( code >= 200 && code < 300 ) + m_access = resp.Response()["access_token"].Str() ; + else + { + Log( "Failed to refresh auth token: HTTP %1%, body: %2%", + code, m_agent->LastError(), log::error ) ; + BOOST_THROW_EXCEPTION( AuthFailed() ); + } } std::string OAuth2::RefreshToken( ) const diff --git a/libgrive/src/protocol/OAuth2.hh b/libgrive/src/protocol/OAuth2.hh index 71c1c014..28e5bc77 100644 --- a/libgrive/src/protocol/OAuth2.hh +++ b/libgrive/src/protocol/OAuth2.hh @@ -20,6 +20,7 @@ #pragma once #include "http/Agent.hh" +#include "util/Exception.hh" #include #include @@ -27,6 +28,9 @@ namespace gr { class OAuth2 { +public : + struct AuthFailed : virtual Exception {} ; + public : OAuth2( std::auto_ptr& agent, From f288c559c6a1c10cdce1064b5b99c7d81c7ffd93 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 7 Oct 2015 01:57:28 +0300 Subject: [PATCH 098/166] Oops. I was thinking auto_ptr is smart_ptr O_o. Sorry. Also fix #33 --- grive/src/main.cc | 6 +++--- libgrive/src/json/ValBuilder.cc | 2 +- libgrive/src/protocol/AuthAgent.cc | 3 +-- libgrive/src/protocol/AuthAgent.hh | 4 ++-- libgrive/src/protocol/OAuth2.cc | 4 ++-- libgrive/src/protocol/OAuth2.hh | 6 +++--- 6 files changed, 12 insertions(+), 13 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index a6efff5d..f0147ba3 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -153,7 +153,7 @@ int Main( int argc, char **argv ) if ( vm.count( "auth" ) ) { - OAuth2 token( http, client_id, client_secret ) ; + OAuth2 token( http.get(), client_id, client_secret ) ; std::cout << "-----------------------\n" @@ -189,8 +189,8 @@ int Main( int argc, char **argv ) return -1; } - OAuth2 token( http, refresh_token, client_id, client_secret ) ; - AuthAgent agent( token, http ) ; + OAuth2 token( http.get(), refresh_token, client_id, client_secret ) ; + AuthAgent agent( token, http.get() ) ; v2::Syncer2 syncer( &agent ); Drive drive( &syncer, config.GetAll() ) ; diff --git a/libgrive/src/json/ValBuilder.cc b/libgrive/src/json/ValBuilder.cc index 7a0068ec..e3b1b0ab 100644 --- a/libgrive/src/json/ValBuilder.cc +++ b/libgrive/src/json/ValBuilder.cc @@ -132,7 +132,7 @@ Val ValBuilder::Result() const if ( !m_ctx.size() ) BOOST_THROW_EXCEPTION( Error() << NoKey_( Val(std::string("")) ) ) ; Val r = m_ctx.top().val; - if ( m_ctx.size() > 0 ) + if ( m_ctx.size() > 1 ) BOOST_THROW_EXCEPTION( Error() << Unexpected_(m_ctx.top().val) ) ; return r; } diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 145c05d4..4e95d0ca 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -31,11 +31,10 @@ namespace gr { using namespace http ; -AuthAgent::AuthAgent( OAuth2& auth, std::auto_ptr real_agent ) : +AuthAgent::AuthAgent( OAuth2& auth, Agent *real_agent ) : m_auth ( auth ), m_agent ( real_agent ) { - assert( m_agent.get() != 0 ) ; } http::ResponseLog* AuthAgent::GetLog() const diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index 4ceeaffc..db65f443 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -34,7 +34,7 @@ namespace gr { class AuthAgent : public http::Agent { public : - AuthAgent( OAuth2& auth, std::auto_ptr real_agent ) ; + AuthAgent( OAuth2& auth, http::Agent* real_agent ) ; http::ResponseLog* GetLog() const ; void SetLog( http::ResponseLog *log ) ; @@ -86,7 +86,7 @@ private : private : OAuth2& m_auth ; - const std::auto_ptr m_agent ; + http::Agent* m_agent ; } ; } // end of namespace diff --git a/libgrive/src/protocol/OAuth2.cc b/libgrive/src/protocol/OAuth2.cc index be355160..cd3dd6ac 100644 --- a/libgrive/src/protocol/OAuth2.cc +++ b/libgrive/src/protocol/OAuth2.cc @@ -33,7 +33,7 @@ namespace gr { const std::string token_url = "https://accounts.google.com/o/oauth2/token" ; OAuth2::OAuth2( - std::auto_ptr& agent, + http::Agent* agent, const std::string& refresh_code, const std::string& client_id, const std::string& client_secret ) : @@ -46,7 +46,7 @@ OAuth2::OAuth2( } OAuth2::OAuth2( - std::auto_ptr& agent, + http::Agent* agent, const std::string& client_id, const std::string& client_secret ) : m_agent( agent ), diff --git a/libgrive/src/protocol/OAuth2.hh b/libgrive/src/protocol/OAuth2.hh index 28e5bc77..e9a23dae 100644 --- a/libgrive/src/protocol/OAuth2.hh +++ b/libgrive/src/protocol/OAuth2.hh @@ -33,11 +33,11 @@ public : public : OAuth2( - std::auto_ptr& agent, + http::Agent* agent, const std::string& client_id, const std::string& client_secret ) ; OAuth2( - std::auto_ptr& agent, + http::Agent* agent, const std::string& refresh_code, const std::string& client_id, const std::string& client_secret ) ; @@ -58,7 +58,7 @@ public : private : std::string m_access ; std::string m_refresh ; - std::auto_ptr m_agent ; + http::Agent* m_agent ; const std::string m_client_id ; const std::string m_client_secret ; From d9300c953e25c145d30ecb894f58a280aa987656 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 8 Oct 2015 01:41:33 +0300 Subject: [PATCH 099/166] Fix .trash and .grive_state in drive location when used with -p option Previously .trash and .grive_state were incorrectly placed in the current working directory instead of synced root directory (-p). --- libgrive/src/base/Drive.cc | 2 +- libgrive/src/base/Resource.cc | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index a8efb8a4..7aa282af 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -85,7 +85,7 @@ void Drive::FromChange( const Entry& entry ) void Drive::SaveState() { - m_state.Write( state_file ) ; + m_state.Write( m_root / state_file ) ; } void Drive::SyncFolders( ) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index e206ebc5..32c77529 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -461,13 +461,20 @@ void Resource::DeleteLocal() static const boost::format trash_file( "%1%-%2%" ) ; assert( m_parent != 0 ) ; - fs::path parent = m_parent->Path() ; - fs::path dest = ".trash" / parent / Name() ; - + Resource* p = m_parent; + fs::path destdir; + while ( !p->IsRoot() ) + { + destdir = p->Name() / destdir; + p = p->Parent(); + } + destdir = p->Path() / ".trash" / destdir; + + fs::path dest = destdir / Name(); std::size_t idx = 1 ; while ( fs::exists( dest ) && idx != 0 ) - dest = ".trash" / parent / (boost::format(trash_file) % Name() % idx++).str() ; - + dest = destdir / (boost::format(trash_file) % Name() % idx++).str() ; + // wrap around! just remove the file if ( idx == 0 ) fs::remove_all( Path() ) ; From 1cca10272d5e03a7dda3fd0e596e4a677cf49e96 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 8 Oct 2015 13:30:50 +0300 Subject: [PATCH 100/166] Track both server-side and local sync times - fixes #6 --- libgrive/src/base/Resource.cc | 14 ++++---- libgrive/src/base/Resource.hh | 6 ++-- libgrive/src/base/ResourceTree.cc | 4 +-- libgrive/src/base/ResourceTree.hh | 2 +- libgrive/src/base/State.cc | 54 +++++++++++++++++-------------- libgrive/src/base/State.hh | 1 + 6 files changed, 44 insertions(+), 37 deletions(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 32c77529..61a82d20 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -71,7 +71,7 @@ void Resource::SetState( State new_state ) boost::bind( &Resource::SetState, _1, new_state ) ) ; } -void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_sync ) +void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_change ) { fs::path path = Path() ; @@ -86,7 +86,7 @@ void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_sync } // remote file created after last sync, so remote is newer - else if ( remote.MTime() > last_sync ) + else if ( remote.MTime() > last_change ) { if ( fs::exists( path ) ) { @@ -120,13 +120,13 @@ void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_sync /// Update the state according to information (i.e. Entry) from remote. This function /// compares the modification time and checksum of both copies and determine which /// one is newer. -void Resource::FromRemote( const Entry& remote, const DateTime& last_sync ) +void Resource::FromRemote( const Entry& remote, const DateTime& last_change ) { // sync folder if ( remote.IsDir() && IsFolder() ) - FromRemoteFolder( remote, last_sync ) ; + FromRemoteFolder( remote, last_change ) ; else - FromRemoteFile( remote, last_sync ) ; + FromRemoteFile( remote, last_change ) ; AssignIDs( remote ) ; @@ -152,7 +152,7 @@ void Resource::AssignIDs( const Entry& remote ) } } -void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_sync ) +void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_change ) { assert( m_parent != 0 ) ; @@ -175,7 +175,7 @@ void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_sync ) { Trace( "file %1% change stamp = %2%", Path(), remote.ChangeStamp() ) ; - if ( remote.MTime() > last_sync || remote.ChangeStamp() > 0 ) + if ( remote.MTime() > last_change || remote.ChangeStamp() > 0 ) { Log( "file %1% is created in remote (change %2%)", path, remote.ChangeStamp(), log::verbose ) ; diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index 06a61a4f..cb712c62 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -106,7 +106,7 @@ public : bool HasID() const ; std::string MD5() const ; - void FromRemote( const Entry& remote, const DateTime& last_sync ) ; + void FromRemote( const Entry& remote, const DateTime& last_change ) ; void FromLocal( const DateTime& last_sync ) ; void Sync( Syncer* syncer, DateTime& sync_time, const Val& options ) ; @@ -128,8 +128,8 @@ private : private : void SetState( State new_state ) ; - void FromRemoteFolder( const Entry& remote, const DateTime& last_sync ) ; - void FromRemoteFile( const Entry& remote, const DateTime& last_sync ) ; + void FromRemoteFolder( const Entry& remote, const DateTime& last_change ) ; + void FromRemoteFile( const Entry& remote, const DateTime& last_change ) ; void DeleteLocal() ; diff --git a/libgrive/src/base/ResourceTree.cc b/libgrive/src/base/ResourceTree.cc index 738b8028..66bb379a 100644 --- a/libgrive/src/base/ResourceTree.cc +++ b/libgrive/src/base/ResourceTree.cc @@ -123,11 +123,11 @@ void ResourceTree::Erase( Resource *coll ) s.erase( s.find( coll ) ) ; } -void ResourceTree::Update( Resource *coll, const Entry& e, const DateTime& last_sync ) +void ResourceTree::Update( Resource *coll, const Entry& e, const DateTime& last_change ) { assert( coll != 0 ) ; - coll->FromRemote( e, last_sync ) ; + coll->FromRemote( e, last_change ) ; ReInsert( coll ) ; } diff --git a/libgrive/src/base/ResourceTree.hh b/libgrive/src/base/ResourceTree.hh index b74d9f19..c1945180 100644 --- a/libgrive/src/base/ResourceTree.hh +++ b/libgrive/src/base/ResourceTree.hh @@ -75,7 +75,7 @@ public : void Insert( Resource *coll ) ; void Erase( Resource *coll ) ; - void Update( Resource *coll, const Entry& e, const DateTime& last_sync ) ; + void Update( Resource *coll, const Entry& e, const DateTime& last_change ) ; Resource* Root() ; const Resource* Root() const ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index fa1e037d..7b64fb46 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -44,9 +44,12 @@ State::State( const fs::path& filename, const Val& options ) : // the "-f" option will make grive always think remote is newer Val force ; if ( options.Get("force", force) && force.Bool() ) - m_last_sync = DateTime() ; + { + m_last_change = DateTime() ; + m_last_sync = DateTime::Now() ; + } - Log( "last sync time: %1%", m_last_sync, log::verbose ) ; + Log( "last server change time: %1%; last sync time: %2%", m_last_change, m_last_sync, log::verbose ) ; } State::~State() @@ -69,7 +72,7 @@ void State::FromLocal( const fs::path& p, Resource* folder ) { assert( folder != 0 ) ; assert( folder->IsFolder() ) ; - + // sync the folder itself folder->FromLocal( m_last_sync ) ; @@ -188,7 +191,7 @@ bool State::Update( const Entry& e ) Log( "%1% is ignored by grive", path, log::verbose ) ; return true; } - m_res.Update( res, e, m_last_sync ) ; + m_res.Update( res, e, m_last_change ) ; } else if ( Resource *parent = m_res.FindByHref( e.ParentHref() ) ) { @@ -207,7 +210,7 @@ bool State::Update( const Entry& e ) if ( child != 0 ) { // since we are updating the ID and Href, we need to remove it and re-add it. - m_res.Update( child, e, m_last_sync ) ; + m_res.Update( child, e, m_last_change ) ; } // folder entry exist in google drive, but not local. we should create @@ -220,7 +223,7 @@ bool State::Update( const Entry& e ) m_res.Insert( child ) ; // update the state of the resource - m_res.Update( child, e, m_last_sync ) ; + m_res.Update( child, e, m_last_change ) ; } return true ; @@ -246,22 +249,22 @@ State::iterator State::end() void State::Read( const fs::path& filename ) { + m_last_sync.Assign( 0 ) ; + m_last_change.Assign( 0 ) ; try { File file( filename ) ; Val json = ParseJson( file ); - Val last_sync = json["last_sync"] ; - m_last_sync.Assign( - last_sync["sec"].Int(), - last_sync["nsec"].Int() ) ; - + Val last_change = json.Has( "last_change" ) ? json["last_change"] : json["last_sync"] ; + m_last_sync.Assign( last_sync["sec"].Int(), last_sync["nsec"].Int() ) ; + m_last_change.Assign( last_change["sec"].Int(), last_change["nsec"].Int() ) ; + m_cstamp = json["change_stamp"].Int() ; } catch ( Exception& ) { - m_last_sync.Assign(0) ; } } @@ -271,8 +274,13 @@ void State::Write( const fs::path& filename ) const last_sync.Add( "sec", Val( (int)m_last_sync.Sec() ) ); last_sync.Add( "nsec", Val( (unsigned)m_last_sync.NanoSec() ) ); + Val last_change ; + last_change.Add( "sec", Val( (int)m_last_change.Sec() ) ); + last_change.Add( "nsec", Val( (unsigned)m_last_change.NanoSec() ) ); + Val result ; result.Add( "last_sync", last_sync ) ; + result.Add( "last_change", last_change ) ; result.Add( "change_stamp", Val(m_cstamp) ) ; std::ofstream fs( filename.string().c_str() ) ; @@ -288,19 +296,17 @@ void State::Sync( Syncer *syncer, const Val& options ) // the last sync time would always be a server time rather than a client time // TODO - WARNING - do we use the last sync time to compare to client file times // need to check if this introduces a new problem - DateTime last_sync_time = m_last_sync; - m_res.Root()->Sync( syncer, last_sync_time, options ) ; + DateTime last_change_time = m_last_change; + m_res.Root()->Sync( syncer, last_change_time, options ) ; - if ( last_sync_time == m_last_sync ) - { - Trace( "nothing changed? %1%", m_last_sync ) ; - m_last_sync = DateTime::Now(); - } - else - { - Trace( "updating last sync? %1%", last_sync_time ) ; - m_last_sync = last_sync_time; - } + if ( last_change_time == m_last_change ) + Trace( "nothing changed at the server side since %1%", m_last_change ) ; + else + { + Trace( "updating last server-side change time to %1%", last_change_time ) ; + m_last_change = last_change_time; + } + m_last_sync = DateTime::Now(); } long State::ChangeStamp() const diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index acbb6994..53814eb7 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -75,6 +75,7 @@ private : private : ResourceTree m_res ; DateTime m_last_sync ; + DateTime m_last_change ; int m_cstamp ; std::string m_dir ; boost::regex m_ign ; From 679fa0eec6f7c450423711f3a952bbdaa4f25eca Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 8 Oct 2015 14:23:59 +0300 Subject: [PATCH 101/166] Use title instead of originalFilename - fixes #32 --- libgrive/src/drive2/Entry2.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index e5db8f06..9e9ce7a5 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -56,7 +56,7 @@ void Entry2::Update( const Val& item ) m_title = file["title"] ; m_etag = file["etag"] ; Val fn; - m_filename = file.Get( "originalFilename", fn ) ? fn.Str() : std::string(); + m_filename = file.Get( "title", fn ) ? fn.Str() : std::string(); m_self_href = file["selfLink"] ; m_mtime = DateTime( file["modifiedDate"] ) ; From 887da88c148847047735e033c53bb41e3f907bcf Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 9 Oct 2015 12:31:19 +0300 Subject: [PATCH 102/166] Simplify http::Agent methods and use multipart upload Should speed-up uploads by updating metadata and contents at the same time. Also should fix most 412 errors because of the update atomicity. --- README.md | 11 ++- libgrive/src/drive/Syncer1.cc | 2 +- libgrive/src/drive2/Syncer2.cc | 69 ++++++++-------- libgrive/src/http/Agent.cc | 67 ++++++++++++++++ libgrive/src/http/Agent.hh | 14 ++-- libgrive/src/http/CurlAgent.cc | 121 ++++------------------------- libgrive/src/http/CurlAgent.hh | 32 ++------ libgrive/src/protocol/AuthAgent.cc | 85 ++------------------ libgrive/src/protocol/AuthAgent.hh | 26 +------ libgrive/src/util/ConcatStream.cc | 93 ++++++++++++++++++++++ libgrive/src/util/ConcatStream.hh | 48 ++++++++++++ libgrive/src/util/DataStream.hh | 11 ++- libgrive/src/util/File.hh | 2 +- libgrive/src/util/StringStream.cc | 40 +++++++--- libgrive/src/util/StringStream.hh | 11 ++- 15 files changed, 340 insertions(+), 292 deletions(-) create mode 100644 libgrive/src/http/Agent.cc create mode 100644 libgrive/src/util/ConcatStream.cc create mode 100644 libgrive/src/util/ConcatStream.hh diff --git a/README.md b/README.md index 676efab9..c5fe251e 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# Grive2 0.4.1 +# Grive2 0.4.2-dev -4 Aug 2015, Vitaliy Filippov +10 Oct 2015, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 @@ -28,7 +28,7 @@ You need the following libraries: - libcurl - libstdc++ - libgcrypt -- Boost (Boost filesystem, program_options and regex are required) +- Boost (Boost filesystem, program_options, regex, unit_test_framework and system are required) - expat There are also some optional dependencies: @@ -62,7 +62,10 @@ Enjoy! ### Grive2 v0.4.2 (unreleased) -- Exclude files by perl regexp +- Option to exclude files by perl regexp +- Reimplemented HTTP response logging for debug purposes +- Use multipart uploads (update metadata and contents at the same time) for improved perfomance & stability +- Bug fixes ### Grive2 v0.4.1 diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index 1374a5e4..9ea8a2d4 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -75,7 +75,7 @@ void Syncer1::DeleteRemote( Resource *res ) m_http->Get( res->SelfHref(), &xml, hdr ) ; AssignIDs( res, Entry1( xml.Response() ) ) ; - m_http->Custom( "DELETE", res->SelfHref(), &str, hdr ) ; + m_http->Request( "DELETE", res->SelfHref(), NULL, &str, hdr ) ; } catch ( Exception& e ) { diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index eb5e943a..f8fc8901 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -32,6 +32,8 @@ #include "util/OS.hh" #include "util/log/Log.hh" +#include "util/StringStream.hh" +#include "util/ConcatStream.hh" #include @@ -87,14 +89,19 @@ bool Syncer2::Create( Resource *res ) return Upload( res ); } +std::string to_string( uint64_t n ) +{ + std::ostringstream s; + s << n; + return s.str(); +} + bool Syncer2::Upload( Resource *res ) { Val meta; meta.Add( "title", Val( res->Name() ) ); if ( res->IsFolder() ) - { meta.Add( "mimeType", Val( mime_types::folder ) ); - } if ( !res->Parent()->IsRoot() ) { Val parent; @@ -107,8 +114,9 @@ bool Syncer2::Upload( Resource *res ) Val valr ; - // Issue metadata update request + if ( res->IsFolder() ) { + // Only issue metadata update request http::Header hdr2 ; hdr2.Add( "Content-Type: application/json" ); http::ValResponse vrsp ; @@ -120,35 +128,34 @@ bool Syncer2::Upload( Resource *res ) valr = vrsp.Response(); assert( !( valr["id"].Str().empty() ) ); } - - if ( !res->IsFolder() ) + else { - while ( true ) - { - File file( res->Path() ) ; - std::ostringstream xcontent_len ; - xcontent_len << "Content-Length: " << file.Size() ; - - http::Header hdr ; - hdr.Add( "Content-Type: application/octet-stream" ) ; - hdr.Add( xcontent_len.str() ) ; - if ( valr.Has( "etag" ) ) - hdr.Add( "If-Match: " + valr["etag"].Str() ) ; - - http::ValResponse vrsp; - long http_code = m_http->Put( upload_base + "/" + valr["id"].Str() + "?uploadType=media", &file, &vrsp, hdr ) ; - if ( http_code == 410 || http_code == 412 ) - { - Log( "request failed with %1%, body: %2%. retrying whole upload in 5s", http_code, m_http->LastError(), log::warning ) ; - os::Sleep( 5 ); - } - else - { - valr = vrsp.Response() ; - assert( !( valr["id"].Str().empty() ) ); - break ; - } - } + File file( res->Path() ) ; + ConcatStream multipart ; + StringStream p1( + "--file_contents\r\nContent-Type: application/json; charset=utf-8\r\n\r\n" + json_meta + + "\r\n--file_contents\r\nContent-Type: application/octet-stream\r\nContent-Length: " + to_string( file.Size() ) + + "\r\n\r\n" + ); + StringStream p2("\r\n--file_contents--\r\n"); + multipart.Append( &p1 ); + multipart.Append( &file ); + multipart.Append( &p2 ); + + http::Header hdr ; + if ( !res->ETag().empty() ) + hdr.Add( "If-Match: " + res->ETag() ) ; + hdr.Add( "Content-Type: multipart/related; boundary=\"file_contents\"" ); + hdr.Add( "Content-Length: " + to_string( multipart.Size() ) ); + + http::ValResponse vrsp; + m_http->Request( + res->ResourceID().empty() ? "POST" : "PUT", + upload_base + ( res->ResourceID().empty() ? "" : "/" + res->ResourceID() ) + "?uploadType=multipart", + &multipart, &vrsp, hdr + ) ; + valr = vrsp.Response() ; + assert( !( valr["id"].Str().empty() ) ); } Entry2 responseEntry = Entry2( valr ) ; diff --git a/libgrive/src/http/Agent.cc b/libgrive/src/http/Agent.cc new file mode 100644 index 00000000..b9143c65 --- /dev/null +++ b/libgrive/src/http/Agent.cc @@ -0,0 +1,67 @@ +/* + Convenience wrapper methods for various kinds of HTTP requests + Copyright (C) 2015 Vitaliy Filippov + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "Agent.hh" +#include "Header.hh" +#include "util/StringStream.hh" + +namespace gr { + +namespace http { + +long Agent::Put( + const std::string& url, + const std::string& data, + DataStream *dest, + const Header& hdr ) +{ + StringStream s( data ); + return Request( "PUT", url, &s, dest, hdr ); +} + +long Agent::Put( + const std::string& url, + File *file, + DataStream *dest, + const Header& hdr ) +{ + return Request( "PUT", url, (SeekStream*)file, dest, hdr ); +} + +long Agent::Get( + const std::string& url, + DataStream *dest, + const Header& hdr ) +{ + return Request( "GET", url, NULL, dest, hdr ); +} + +long Agent::Post( + const std::string& url, + const std::string& data, + DataStream *dest, + const Header& hdr ) +{ + Header h( hdr ) ; + StringStream s( data ); + h.Add( "Content-Type: application/x-www-form-urlencoded" ); + return Request( "POST", url, &s, dest, h ); +} + +} } // end of namespace diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index 09d3b87d..cb8594af 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -21,10 +21,11 @@ #include #include "ResponseLog.hh" +#include "util/Types.hh" namespace gr { -class DataStream ; +class SeekStream ; class File ; namespace http { @@ -43,28 +44,29 @@ public : const std::string& url, const std::string& data, DataStream *dest, - const Header& hdr ) = 0 ; + const Header& hdr ) ; virtual long Put( const std::string& url, File *file, DataStream *dest, - const Header& hdr ) = 0 ; + const Header& hdr ) ; virtual long Get( const std::string& url, DataStream *dest, - const Header& hdr ) = 0 ; + const Header& hdr ) ; virtual long Post( const std::string& url, const std::string& data, DataStream *dest, - const Header& hdr ) = 0 ; + const Header& hdr ) ; - virtual long Custom( + virtual long Request( const std::string& method, const std::string& url, + SeekStream *in, DataStream *dest, const Header& hdr ) = 0 ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 2e9d5668..a527d519 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -46,35 +46,15 @@ namespace { using namespace gr::http ; using namespace gr ; -std::size_t ReadStringCallback( void *ptr, std::size_t size, std::size_t nmemb, std::string *data ) -{ - assert( ptr != 0 ) ; - assert( data != 0 ) ; - - std::size_t count = std::min( size * nmemb, data->size() ) ; - if ( count > 0 ) - { - std::memcpy( ptr, &(*data)[0], count ) ; - data->erase( 0, count ) ; - } - - return count ; -} - -std::size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, File *file ) +std::size_t ReadFileCallback( void *ptr, std::size_t size, std::size_t nmemb, SeekStream *file ) { assert( ptr != 0 ) ; assert( file != 0 ) ; - std::size_t count = std::min( - static_cast(size * nmemb), - static_cast(file->Size() - file->Tell()) ) ; - assert( count <= std::numeric_limits::max() ) ; - - if ( count > 0 ) - file->Read( static_cast(ptr), static_cast(count) ) ; - - return count ; + if ( size*nmemb > 0 ) + return file->Read( static_cast(ptr), size*nmemb ) ; + + return 0 ; } } // end of local namespace @@ -214,96 +194,27 @@ long CurlAgent::ExecCurl( return http_code ; } -long CurlAgent::Put( - const std::string& url, - const std::string& data, - DataStream *dest, - const Header& hdr ) -{ - Trace("HTTP PUT \"%1%\"", url ) ; - - Init() ; - CURL *curl = m_pimpl->curl ; - - std::string put_data = data ; - - // set common options - ::curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L ) ; - ::curl_easy_setopt(curl, CURLOPT_READFUNCTION, &ReadStringCallback ) ; - ::curl_easy_setopt(curl, CURLOPT_READDATA , &put_data ) ; - ::curl_easy_setopt(curl, CURLOPT_INFILESIZE, put_data.size() ) ; - - return ExecCurl( url, dest, hdr ) ; -} - -long CurlAgent::Put( +long CurlAgent::Request( + const std::string& method, const std::string& url, - File *file, + SeekStream *in, DataStream *dest, const Header& hdr ) { - assert( file != 0 ) ; + Trace("HTTP %1% \"%2%\"", method, url ) ; - Trace("HTTP PUT \"%1%\"", url ) ; - Init() ; CURL *curl = m_pimpl->curl ; // set common options - ::curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L ) ; - ::curl_easy_setopt(curl, CURLOPT_READFUNCTION, &ReadFileCallback ) ; - ::curl_easy_setopt(curl, CURLOPT_READDATA , file ) ; - ::curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, static_cast(file->Size()) ) ; - - return ExecCurl( url, dest, hdr ) ; -} - -long CurlAgent::Get( - const std::string& url, - DataStream *dest, - const Header& hdr ) -{ - Trace("HTTP GET \"%1%\"", url ) ; - Init() ; - - // set get specific options - ::curl_easy_setopt(m_pimpl->curl, CURLOPT_HTTPGET, 1L); - - return ExecCurl( url, dest, hdr ) ; -} - -long CurlAgent::Post( - const std::string& url, - const std::string& post_data, - DataStream *dest, - const Header& hdr ) -{ - Trace("HTTP POST \"%1%\" with \"%2%\"", url, post_data ) ; - - Init() ; - CURL *curl = m_pimpl->curl ; - - // set post specific options - ::curl_easy_setopt(curl, CURLOPT_POST, 1L); - ::curl_easy_setopt(curl, CURLOPT_POSTFIELDS, &post_data[0] ) ; - ::curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, post_data.size() ) ; - - return ExecCurl( url, dest, hdr ) ; -} - -long CurlAgent::Custom( - const std::string& method, - const std::string& url, - DataStream *dest, - const Header& hdr ) -{ - Trace("HTTP %2% \"%1%\"", url, method ) ; - - Init() ; - CURL *curl = m_pimpl->curl ; - ::curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, method.c_str() ); -// ::curl_easy_setopt(curl, CURLOPT_VERBOSE, 1 ); + if ( in ) + { + ::curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L ) ; + ::curl_easy_setopt(curl, CURLOPT_READFUNCTION, &ReadFileCallback ) ; + ::curl_easy_setopt(curl, CURLOPT_READDATA , in ) ; + ::curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, static_cast( in->Size() ) ) ; + } return ExecCurl( url, dest, hdr ) ; } diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index 044e2bc0..ac7fe5b1 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -44,35 +44,13 @@ public : ResponseLog* GetLog() const ; void SetLog( ResponseLog *log ) ; - long Put( - const std::string& url, - const std::string& post_data, - DataStream *dest, - const Header& hdr ) ; - - long Put( - const std::string& url, - File *file, - DataStream *dest, - const Header& hdr ) ; - - long Get( - const std::string& url, - DataStream *dest, - const Header& hdr ) ; - - long Post( - const std::string& url, - const std::string& data, - DataStream *dest, - const Header& hdr ) ; - - long Custom( + long Request( const std::string& method, const std::string& url, + SeekStream *in, DataStream *dest, const Header& hdr ) ; - + std::string LastError() const ; std::string LastErrorHeaders() const ; @@ -84,14 +62,14 @@ public : private : static std::size_t HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; static std::size_t Receive( void* ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; - + long ExecCurl( const std::string& url, DataStream *dest, const Header& hdr ) ; void Init() ; - + private : struct Impl ; std::auto_ptr m_pimpl ; diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 4e95d0ca..77cb54f0 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -47,98 +47,29 @@ void AuthAgent::SetLog( http::ResponseLog *log ) return m_agent->SetLog( log ); } -Header AuthAgent::AppendHeader( const Header& hdr ) const +http::Header AuthAgent::AppendHeader( const http::Header& hdr ) const { - Header h(hdr) ; + http::Header h(hdr) ; h.Add( "Authorization: Bearer " + m_auth.AccessToken() ) ; h.Add( "GData-Version: 3.0" ) ; return h ; } -long AuthAgent::Put( - const std::string& url, - const std::string& data, - DataStream *dest, - const Header& hdr ) -{ - long response; - Header auth; - do - { - auth = AppendHeader( hdr ); - response = m_agent->Put( url, data, dest, auth ); - } while ( CheckRetry( response ) ); - return CheckHttpResponse( response, url, auth ); -} - -long AuthAgent::Put( - const std::string& url, - File *file, - DataStream *dest, - const Header& hdr ) -{ - long response; - Header auth; - while ( true ) - { - auth = AppendHeader( hdr ); - response = m_agent->Put( url, file, dest, auth ); - if ( !CheckRetry( response ) ) - break; - file->Seek( 0, SEEK_SET ); - } - - // On 410 Gone or 412 Precondition failed, recovery may be possible so don't - // throw an exception - if ( response == 410 || response == 412 ) - return response; - - return CheckHttpResponse( response, url, auth ); -} - -long AuthAgent::Get( - const std::string& url, - DataStream *dest, - const Header& hdr ) -{ - long response; - Header auth; - do - { - auth = AppendHeader( hdr ); - response = m_agent->Get( url, dest, auth ); - } while ( CheckRetry( response ) ); - return CheckHttpResponse( response, url, auth ); -} - -long AuthAgent::Post( - const std::string& url, - const std::string& data, - DataStream *dest, - const Header& hdr ) -{ - long response; - Header auth; - do - { - auth = AppendHeader( hdr ); - response = m_agent->Post( url, data, dest, auth ); - } while ( CheckRetry( response ) ); - return CheckHttpResponse( response, url, auth ); -} - -long AuthAgent::Custom( +long AuthAgent::Request( const std::string& method, const std::string& url, + SeekStream *in, DataStream *dest, - const Header& hdr ) + const http::Header& hdr ) { long response; Header auth; do { auth = AppendHeader( hdr ); - response = m_agent->Custom( method, url, dest, auth ); + if ( in ) + in->Seek( 0, 0 ); + response = m_agent->Request( method, url, in, dest, auth ); } while ( CheckRetry( response ) ); return CheckHttpResponse( response, url, auth ); } diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index db65f443..3dd3402a 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -39,32 +39,10 @@ public : http::ResponseLog* GetLog() const ; void SetLog( http::ResponseLog *log ) ; - long Put( - const std::string& url, - const std::string& data, - DataStream *dest, - const http::Header& hdr ) ; - - long Put( - const std::string& url, - File* file, - DataStream *dest, - const http::Header& hdr ) ; - - long Get( - const std::string& url, - DataStream *dest, - const http::Header& hdr ) ; - - long Post( - const std::string& url, - const std::string& data, - DataStream *dest, - const http::Header& hdr ) ; - - long Custom( + long Request( const std::string& method, const std::string& url, + SeekStream *in, DataStream *dest, const http::Header& hdr ) ; diff --git a/libgrive/src/util/ConcatStream.cc b/libgrive/src/util/ConcatStream.cc new file mode 100644 index 00000000..e3c4e1dd --- /dev/null +++ b/libgrive/src/util/ConcatStream.cc @@ -0,0 +1,93 @@ +/* + A stream representing a concatenation of several underlying streams + Copyright (C) 2015 Vitaliy Filippov + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#include "ConcatStream.hh" + +namespace gr { + +ConcatStream::ConcatStream() : + m_cur( 0 ), m_size( 0 ), m_pos( 0 ) +{ +} + +std::size_t ConcatStream::Read( char *data, std::size_t size ) +{ + std::size_t done = 0, l; + while ( done < size && m_cur < m_streams.size() ) + { + l = m_streams[m_cur]->Read( data+done, size-done ); + if ( !l ) + { + m_cur++; + if ( m_cur < m_streams.size() ) + m_streams[m_cur]->Seek( 0, 0 ); + } + done += l; + m_pos += l; + } + return done ; +} + +std::size_t ConcatStream::Write( const char *data, std::size_t size ) +{ + return 0 ; +} + +off_t ConcatStream::Seek( off_t offset, int whence ) +{ + if ( whence == 1 ) + offset += m_pos; + else if ( whence == 2 ) + offset += Size(); + if ( offset > Size() ) + offset = Size(); + m_cur = 0; + m_pos = offset; + if ( m_streams.size() ) + { + while ( offset > m_streams[m_cur]->Size() ) + { + offset -= m_streams[m_cur]->Size(); + m_cur++; + } + m_streams[m_cur]->Seek( offset, 0 ); + } + return m_pos ; +} + +off_t ConcatStream::Tell() const +{ + return m_pos ; +} + +u64_t ConcatStream::Size() const +{ + return m_size ; +} + +void ConcatStream::Append( SeekStream *stream ) +{ + if ( stream ) + { + m_streams.push_back( stream ); + m_size += stream->Size(); + } +} + +} // end of namespace diff --git a/libgrive/src/util/ConcatStream.hh b/libgrive/src/util/ConcatStream.hh new file mode 100644 index 00000000..82f71236 --- /dev/null +++ b/libgrive/src/util/ConcatStream.hh @@ -0,0 +1,48 @@ +/* + A stream representing a concatenation of several underlying streams + Copyright (C) 2015 Vitaliy Filippov + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License + as published by the Free Software Foundation version 2 + of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +*/ + +#pragma once + +#include "DataStream.hh" + +#include + +namespace gr { + +class ConcatStream : public SeekStream +{ +public : + ConcatStream() ; + + std::size_t Read( char *data, std::size_t size ) ; + std::size_t Write( const char *data, std::size_t size ) ; + + off_t Seek( off_t offset, int whence ) ; + off_t Tell() const ; + u64_t Size() const ; + + void Append( SeekStream *stream ) ; + +private : + std::vector m_streams ; + off_t m_size, m_pos ; + int m_cur ; +} ; + +} // end of namespace diff --git a/libgrive/src/util/DataStream.hh b/libgrive/src/util/DataStream.hh index 819c8f8a..1fd06095 100644 --- a/libgrive/src/util/DataStream.hh +++ b/libgrive/src/util/DataStream.hh @@ -20,6 +20,7 @@ #pragma once #include +#include "util/Types.hh" namespace gr { @@ -47,9 +48,13 @@ public : virtual std::size_t Write( const char *data, std::size_t size ) = 0 ; } ; -/// Stream for /dev/null, i.e. read and writing nothing -DataStream* DevNull() ; - +class SeekStream: public DataStream +{ +public : + virtual off_t Seek( off_t offset, int whence ) = 0 ; + virtual off_t Tell() const = 0 ; + virtual u64_t Size() const = 0 ; +} ; } // end of namespace diff --git a/libgrive/src/util/File.hh b/libgrive/src/util/File.hh index 041c5998..d756f484 100644 --- a/libgrive/src/util/File.hh +++ b/libgrive/src/util/File.hh @@ -35,7 +35,7 @@ namespace gr { It is a simple wrapper around the UNIX file descriptor. It will throw exceptions (i.e. Error) when it encounters errors. */ -class File : public DataStream +class File : public SeekStream { public : /// File specific errors. It often includes diff --git a/libgrive/src/util/StringStream.cc b/libgrive/src/util/StringStream.cc index 2cefdfed..755459b9 100644 --- a/libgrive/src/util/StringStream.cc +++ b/libgrive/src/util/StringStream.cc @@ -27,33 +27,55 @@ namespace { // the max size of the cached string. this is to prevent allocating // too much memory if client sends a line too long (i.e. DOS attack) - const std::size_t capacity = 1024 ; + const std::size_t capacity = 4096 ; } StringStream::StringStream( const std::string& init ) : - m_str( init ) + // FIXME avoid copy + m_str( init ), m_pos( 0 ) { } -/// Read `size` bytes from the stream. Those bytes will be removed from -/// the underlying string by calling `std::string::erase()`. Therefore, it is -/// not a good idea to call Read() to read byte-by-byte. +/// Read `size` bytes from the stream. std::size_t StringStream::Read( char *data, std::size_t size ) { // wow! no need to count count == 0 - std::size_t count = std::min( m_str.size(), size ) ; - std::copy( m_str.begin(), m_str.begin() + count, data ) ; - m_str.erase( 0, count ) ; + std::size_t count = std::min( m_str.size()-m_pos, size ) ; + std::copy( m_str.begin() + m_pos, m_str.begin() + m_pos + count, data ) ; + m_pos += count ; return count ; } std::size_t StringStream::Write( const char *data, std::size_t size ) { std::size_t count = std::min( size, capacity - m_str.size() ) ; - m_str.insert( m_str.end(), data, data+count ) ; + m_str.replace( m_str.begin() + m_pos, m_str.begin() + m_pos + count, data, data+count ) ; + m_pos += count ; return count ; } +off_t StringStream::Seek( off_t offset, int whence ) +{ + if ( whence == 1 ) + offset += m_pos; + else if ( whence == 2 ) + offset += Size(); + if ( offset > Size() ) + offset = Size(); + m_pos = (size_t)offset; + return m_pos; +} + +off_t StringStream::Tell() const +{ + return m_pos; +} + +u64_t StringStream::Size() const +{ + return m_str.size(); +} + const std::string& StringStream::Str() const { return m_str ; diff --git a/libgrive/src/util/StringStream.hh b/libgrive/src/util/StringStream.hh index 600f24c3..acf96c13 100644 --- a/libgrive/src/util/StringStream.hh +++ b/libgrive/src/util/StringStream.hh @@ -35,10 +35,8 @@ namespace gr { is to prevent DOS attacks in which the client sends a lot of bytes before the delimiter (e.g. new-line characters) and the server is forced to hold all of them in memory. - - The limit is current 1024 bytes. */ -class StringStream : public DataStream +class StringStream : public SeekStream { public : explicit StringStream( const std::string& init = std::string() ) ; @@ -46,11 +44,16 @@ public : std::size_t Read( char *data, std::size_t size ) ; std::size_t Write( const char *data, std::size_t size ) ; + off_t Seek( off_t offset, int whence ) ; + off_t Tell() const ; + u64_t Size() const ; + const std::string& Str() const ; void Str( const std::string& str ) ; - + private : std::string m_str ; + std::size_t m_pos ; } ; } // end of namespace From df99954382a186715d5eb1bdd0a91a9c65bf2aea Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 10 Oct 2015 13:00:54 +0300 Subject: [PATCH 103/166] Change m_str.replace back to insert + remove StringStream limit --- libgrive/src/util/StringStream.cc | 13 ++----------- libgrive/src/util/StringStream.hh | 5 ----- 2 files changed, 2 insertions(+), 16 deletions(-) diff --git a/libgrive/src/util/StringStream.cc b/libgrive/src/util/StringStream.cc index 755459b9..a128f746 100644 --- a/libgrive/src/util/StringStream.cc +++ b/libgrive/src/util/StringStream.cc @@ -23,13 +23,6 @@ namespace gr { -namespace -{ - // the max size of the cached string. this is to prevent allocating - // too much memory if client sends a line too long (i.e. DOS attack) - const std::size_t capacity = 4096 ; -} - StringStream::StringStream( const std::string& init ) : // FIXME avoid copy m_str( init ), m_pos( 0 ) @@ -48,10 +41,8 @@ std::size_t StringStream::Read( char *data, std::size_t size ) std::size_t StringStream::Write( const char *data, std::size_t size ) { - std::size_t count = std::min( size, capacity - m_str.size() ) ; - m_str.replace( m_str.begin() + m_pos, m_str.begin() + m_pos + count, data, data+count ) ; - m_pos += count ; - return count ; + m_str.insert( m_str.size(), data, size ) ; + return size ; } off_t StringStream::Seek( off_t offset, int whence ) diff --git a/libgrive/src/util/StringStream.hh b/libgrive/src/util/StringStream.hh index acf96c13..39d4fa1f 100644 --- a/libgrive/src/util/StringStream.hh +++ b/libgrive/src/util/StringStream.hh @@ -30,11 +30,6 @@ namespace gr { StringStream is a DataStream back-end that uses std::string for storage. It is useful for unit tests and text parsing, especially when used with StreamParser. - - StringStream has a limit on the maximum number of byte it stores. This - is to prevent DOS attacks in which the client sends a lot of bytes before - the delimiter (e.g. new-line characters) and the server is forced to hold - all of them in memory. */ class StringStream : public SeekStream { From 81b1a4f05c228fdbec0d534a8f3ef3e680f88a40 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 10 Oct 2015 21:43:54 +0000 Subject: [PATCH 104/166] Fix non-virtual destructor and missing return value (fixes "illegal instruction" under FreeBSD) --- libgrive/src/base/State.cc | 1 + libgrive/src/util/log/Log.hh | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 7b64fb46..fed5f157 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -192,6 +192,7 @@ bool State::Update( const Entry& e ) return true; } m_res.Update( res, e, m_last_change ) ; + return true; } else if ( Resource *parent = m_res.FindByHref( e.ParentHref() ) ) { diff --git a/libgrive/src/util/log/Log.hh b/libgrive/src/util/log/Log.hh index 0d43adbb..9646042f 100644 --- a/libgrive/src/util/log/Log.hh +++ b/libgrive/src/util/log/Log.hh @@ -66,7 +66,7 @@ public : virtual bool IsEnabled( log::Serverity s ) const = 0 ; static LogBase* Inst( std::auto_ptr log = std::auto_ptr() ) ; - ~LogBase() ; + virtual ~LogBase() ; protected : LogBase() ; From f0b6bcf8b4769ef10f1f7e5ce9723beb391cdd44 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 11 Oct 2015 15:30:24 +0300 Subject: [PATCH 105/166] Add Debian packaging scripts Based on work by Alin Andrei (https://launchpad.net/~nilarimogard/+archive/ubuntu/webupd8) --- debian/changelog | 78 ++++++++++++++++++++++++++++++++++++++++++++ debian/compat | 1 + debian/control | 25 ++++++++++++++ debian/copyright | 20 ++++++++++++ debian/rules | 4 +++ debian/source/format | 1 + 6 files changed, 129 insertions(+) create mode 100644 debian/changelog create mode 100644 debian/compat create mode 100644 debian/control create mode 100644 debian/copyright create mode 100755 debian/rules create mode 100644 debian/source/format diff --git a/debian/changelog b/debian/changelog new file mode 100644 index 00000000..7ddc1e34 --- /dev/null +++ b/debian/changelog @@ -0,0 +1,78 @@ +grive2 (0.4.1+git20151011) unstable; urgency=medium + + * Add Debian packaging scripts to the official repository + + -- Vitaliy Filippov Sun, 11 Oct 2015 15:03:55 +0300 + +grive2 (0.4.1-1+git20151008~webupd8~wily0) wily; urgency=medium + + * new git pull + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Thu, 08 Oct 2015 11:50:55 +0200 + +grive2 (0.4.1-1+git20151007~webupd8~wily0) wily; urgency=medium + + * new git pull + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Wed, 07 Oct 2015 13:46:29 +0200 + +grive2 (0.4.1-1+git20151001~webupd8~precise0) precise; urgency=medium + + * new git pull + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Thu, 01 Oct 2015 12:54:44 +0200 + +grive2 (0.4.0-1+git20150928~webupd8~wily1) wily; urgency=medium + + * new git pull + * build-depend on libboost-regex-dev + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Mon, 28 Sep 2015 11:36:19 +0200 + +grive2 (0.4.0-1+git20150810~webupd8~wily0) wily; urgency=medium + + * upload for Wily + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Sat, 04 Jul 2015 12:26:14 +0200 + +grive2 (0.4.0-1+git20150810~webupd8~vivid0) vivid; urgency=medium + + * new git pull + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Sat, 04 Jul 2015 12:26:14 +0200 + +grive2 (0.4.0-1+git20150629~webupd8~vivid0) vivid; urgency=medium + + * new git pull + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Mon, 29 Jun 2015 12:27:21 +0200 + +grive2 (0.4.0-1~webupd8~precise3) precise; urgency=medium + + * Initial packaging (based on grive) + * For Precise and Trusty only: use libgcrypt11 instead of libgcrypt20 + because libcurl4-openssl-dev depends on it (and for Precise, + there's no libgcrypt20) + + -- Alin Andrei Mon, 25 May 2015 15:31:24 +0200 diff --git a/debian/compat b/debian/compat new file mode 100644 index 00000000..7f8f011e --- /dev/null +++ b/debian/compat @@ -0,0 +1 @@ +7 diff --git a/debian/control b/debian/control new file mode 100644 index 00000000..2b658956 --- /dev/null +++ b/debian/control @@ -0,0 +1,25 @@ +Source: grive2 +Section: net +Priority: optional +Maintainer: Vitaliy Filippov +Build-Depends: debhelper, cmake, pkg-config, zlib1g-dev, libcurl4-openssl-dev, libstdc++6-4.4-dev | libstdc++-4.9-dev | libstdc++-5-dev, libboost-filesystem-dev, libboost-program-options-dev, libboost-test-dev, libboost-regex-dev, libexpat1-dev, binutils-dev, libgcrypt-dev, libyajl-dev +Standards-Version: 3.9.6 +Homepage: https://yourcmc.ru/wiki/Grive2 + +Package: grive +Architecture: any +Depends: ${shlibs:Depends}, ${misc:Depends} +Description: Grive2: an open source Linux client for Google Drive + . + This is the up-to-date fork of the original "Grive" (https://github.com/Grive/grive) + Google Drive client with the support for the new Drive REST API and partial + sync. + . + For the first time running grive, you should use the "-a" argument to grant + permission to grive to access to your Google Drive. An URL should be printed. + Go to the page. You will need to login to your google account if you haven't + done so. After granting the permission to grive, the browser will show you + an authenication code. Copy-and-paste that to the standard input of grive. + If everything works fine, grive will create .grive and .grive_state inside the + synchronized directory. It will also start downloading files from your Google + Drive to that directory. diff --git a/debian/copyright b/debian/copyright new file mode 100644 index 00000000..e3628400 --- /dev/null +++ b/debian/copyright @@ -0,0 +1,20 @@ +Grive2 + + https://github.com/vitalif/grive2 + +Current developers: + + Vitaliy Filippov + +Previous developers: + + Nestal Wan (me@nestal.net) 16.05.2012 — 03.05.2013 + Matchman Green (match065@gmail.com) 26.04.2012 — 20.06.2012 + +Contributors: + + See full list here https://yourcmc.ru/wiki/Grive2#Full_list_of_contributors + +License: + + GPL 2.0+ diff --git a/debian/rules b/debian/rules new file mode 100755 index 00000000..81484573 --- /dev/null +++ b/debian/rules @@ -0,0 +1,4 @@ +#!/usr/bin/make -f + +%: + dh $@ --buildsystem=cmake --parallel diff --git a/debian/source/format b/debian/source/format new file mode 100644 index 00000000..89ae9db8 --- /dev/null +++ b/debian/source/format @@ -0,0 +1 @@ +3.0 (native) From 44cb91f94e67bcdad77ca564b4d8083cedf72251 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 11 Oct 2015 18:59:25 +0300 Subject: [PATCH 106/166] State that YAJL 2.x is required --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c5fe251e..1522d016 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ These may be added in the future, possibly the next release. You need the following libraries: -- yajl +- yajl 2.x - libcurl - libstdc++ - libgcrypt From 9156785334abad548b9f2190a820f3f2fb3990f3 Mon Sep 17 00:00:00 2001 From: Teddy Reed Date: Mon, 9 Nov 2015 20:27:55 -0800 Subject: [PATCH 107/166] Check for missing downloadUrl --- libgrive/src/drive2/Entry2.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index 9e9ce7a5..75f7d96f 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -66,7 +66,7 @@ void Entry2::Update( const Val& item ) m_is_removed = file["labels"]["trashed"].Bool() ; if ( !m_is_dir ) { - if ( !file.Has( "md5Checksum" ) ) + if ( !file.Has( "md5Checksum" ) || !file.Has("downloadUrl") ) { // This is either a google docs document or a not-yet-uploaded file. Ignore it. // FIXME: We'll need to compare timestamps to support google docs. From 3b608329b9c66583a0f985c4a4393af5dafc293e Mon Sep 17 00:00:00 2001 From: Dylan Wulf Date: Tue, 10 Nov 2015 12:54:47 +0300 Subject: [PATCH 108/166] Added ability to move and rename files/directories --- grive/src/main.cc | 16 +++++++++ libgrive/src/base/Drive.cc | 5 +++ libgrive/src/base/Drive.hh | 1 + libgrive/src/base/State.cc | 63 ++++++++++++++++++++++++++++++++++ libgrive/src/base/State.hh | 3 +- libgrive/src/base/Syncer.hh | 1 + libgrive/src/drive2/Syncer2.cc | 41 ++++++++++++++++++++++ libgrive/src/drive2/Syncer2.hh | 1 + 8 files changed, 130 insertions(+), 1 deletion(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index f0147ba3..0472b1a5 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -121,6 +121,7 @@ int Main( int argc, char **argv ) ( "dry-run", "Only detect which files need to be uploaded/downloaded, " "without actually performing them." ) ( "ignore", po::value(), "Ignore files relative paths of which match this Perl RegExp." ) + ( "move,m", po::value >()->multitoken(), "Syncs, then moves a file (first argument) to new location (second argument) without reuploading or redownloading." ) ; po::variables_map vm; @@ -203,6 +204,21 @@ int Main( int argc, char **argv ) } else drive.DryRun() ; + + if ( vm.count ( "move" ) > 0 && vm.count( "dry-run" ) == 0 ) + { + if (vm["move"].as >().size() < 2 ) + Log( "Not enough arguments for move. Move failed.", log::error ); + else + { + bool success = drive.Move( vm["move"].as >()[0], + vm["move"].as >()[1] ); + if (success) + Log( "Move successful!", log::info ); + else + Log( "Move failed.", log::error); + } + } config.Save() ; Log( "Finished!", log::info ) ; diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index 7aa282af..63128a17 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -150,6 +150,11 @@ void Drive::DetectChanges() } } +bool Drive::Move( fs::path old_p, fs::path new_p ) +{ + return m_state.Move( m_syncer, old_p, new_p, m_options["path"].Str() ); +} + void Drive::Update() { Log( "Synchronizing files", log::info ) ; diff --git a/libgrive/src/base/Drive.hh b/libgrive/src/base/Drive.hh index 7a63999f..db1a027c 100644 --- a/libgrive/src/base/Drive.hh +++ b/libgrive/src/base/Drive.hh @@ -41,6 +41,7 @@ public : Drive( Syncer *syncer, const Val& options ) ; void DetectChanges() ; + bool Move( fs::path old_p, fs::path new_p ); void Update() ; void DryRun() ; void SaveState() ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index fed5f157..bec157e2 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -321,4 +321,67 @@ void State::ChangeStamp( long cstamp ) m_cstamp = cstamp ; } +bool State::Move( Syncer* syncer, fs::path old_p, fs::path new_p, fs::path grive_root ) +{ + //Convert paths to canonical representations + //Also seems to remove trailing / at the end of directory paths + old_p = fs::canonical( old_p ); + grive_root = fs::canonical( grive_root ); + + //new_p is a little special because fs::canonical() requires that the path exists + if ( new_p.string()[ new_p.string().size() - 1 ] == '/') //If new_p ends with a /, remove it + new_p = new_p.parent_path(); + new_p = fs::canonical( new_p.parent_path() ) / new_p.filename(); + + //Fails if source file doesn't exist, or if destination file already + //exists and is not a directory, or if the source and destination are exactly the same + if ( (fs::exists(new_p) && !fs::is_directory(new_p) ) || !fs::exists(old_p) || fs::equivalent( old_p, new_p ) ) + return false; + + //If new path is an existing directory, move the file into the directory + //instead of trying to rename it + if ( fs::is_directory(new_p) ){ + new_p = new_p / old_p.filename(); + } + + //Get the paths relative to grive root. + //Just finds the substring from the end of the grive_root to the end of the path + //+1s are to exclude slash at beginning of relative path + int start = grive_root.string().size() + 1; + int nLen = new_p.string().size() - (grive_root.string().size() + 1); + int oLen = old_p.string().size() - (grive_root.string().size() + 1); + if ( start + nLen != new_p.string().size() || start + oLen != old_p.string().size() ) + return false; + fs::path new_p_rootrel( new_p.string().substr( start, nLen ) ); + fs::path old_p_rootrel( old_p.string().substr( start, oLen ) ); + + //Get resources + Resource* res = m_res.Root(); + Resource* newParentRes = m_res.Root(); + for ( fs::path::iterator it = old_p_rootrel.begin(); it != old_p_rootrel.end(); ++it ) + { + if ( *it != "." && *it != ".." && res != 0 ) + res = res->FindChild(it->string()); + if ( *it == ".." ) + res = res->Parent(); + } + for ( fs::path::iterator it = new_p_rootrel.begin(); it != new_p_rootrel.end(); ++it ) + { + if ( *it != "." && *it != ".." && *it != new_p.filename() && newParentRes != 0 ) + newParentRes = newParentRes->FindChild(it->string()); + if ( *it == "..") + res = res->Parent(); + } + + //These conditions should only occur if everything is not up-to-date + if ( res == 0 || newParentRes == 0 || res->GetState() != Resource::sync || + newParentRes->GetState() != Resource::sync || + newParentRes->FindChild( new_p.filename().string() ) != 0 ) + return false; + + fs::rename(old_p, new_p); //Moves local file + syncer->Move(res, newParentRes, new_p.filename().string()); //Moves server file + return true; +} + } // end of namespace gr diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 53814eb7..7822be97 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -63,7 +63,8 @@ public : long ChangeStamp() const ; void ChangeStamp( long cstamp ) ; - + bool Move( Syncer* syncer, fs::path old_p, fs::path new_p, fs::path grive_root ); + private : void FromLocal( const fs::path& p, Resource *folder ) ; void FromChange( const Entry& e ) ; diff --git a/libgrive/src/base/Syncer.hh b/libgrive/src/base/Syncer.hh index 33375eee..19adacb5 100644 --- a/libgrive/src/base/Syncer.hh +++ b/libgrive/src/base/Syncer.hh @@ -53,6 +53,7 @@ public : virtual void Download( Resource *res, const fs::path& file ); virtual bool EditContent( Resource *res, bool new_rev ) = 0; virtual bool Create( Resource *res ) = 0; + virtual bool Move( Resource* res, Resource* newParent, std::string newFilename ) = 0; virtual std::auto_ptr GetFolders() = 0; virtual std::auto_ptr GetAll() = 0; diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index f8fc8901..4c0f290b 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -89,6 +89,47 @@ bool Syncer2::Create( Resource *res ) return Upload( res ); } +bool Syncer2::Move( Resource* res, Resource* newParentRes, std::string newFilename ) +{ + if ( res->ResourceID().empty() ) + { + Log("Can't rename file %1%, no server id found", res->Name()); + return false; + } + + Val meta; + meta.Add( "title", Val(newFilename) ); + if ( res->IsFolder() ) + { + meta.Add( "mimeType", Val( mime_types::folder ) ); + } + std::string json_meta = WriteJson( meta ); + + Val valr ; + + // Issue metadata update request + { + std::string addRemoveParents(""); + if (res->Parent()->IsRoot() ) + addRemoveParents += "&removeParents=root"; + else + addRemoveParents += "&removeParents=" + res->Parent()->ResourceID(); + if ( newParentRes->IsRoot() ) + addRemoveParents += "&addParents=root"; + else + addRemoveParents += "&addParents=" + newParentRes->ResourceID(); + http::Header hdr2 ; + hdr2.Add( "Content-Type: application/json" ); + http::ValResponse vrsp ; + long http_code = 0; + //Don't change modified date because we're only moving + http_code = m_http->Put( feeds::files + "/" + res->ResourceID() + "?modifiedDateBehavior=noChange" + addRemoveParents, json_meta, &vrsp, hdr2 ) ; + valr = vrsp.Response(); + assert( !( valr["id"].Str().empty() ) ); + } + return true; +} + std::string to_string( uint64_t n ) { std::ostringstream s; diff --git a/libgrive/src/drive2/Syncer2.hh b/libgrive/src/drive2/Syncer2.hh index 0ab6286c..96f93fce 100644 --- a/libgrive/src/drive2/Syncer2.hh +++ b/libgrive/src/drive2/Syncer2.hh @@ -37,6 +37,7 @@ public : void DeleteRemote( Resource *res ); bool EditContent( Resource *res, bool new_rev ); bool Create( Resource *res ); + bool Move( Resource* res, Resource* newParent, std::string newFilename ); std::auto_ptr GetFolders(); std::auto_ptr GetAll(); From 450ff5857ef72ba66aa335a94c4246cf3c971a06 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Tue, 10 Nov 2015 12:58:55 +0300 Subject: [PATCH 109/166] Add move/rename to README --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 1522d016..9edec32e 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,7 @@ Enjoy! - Reimplemented HTTP response logging for debug purposes - Use multipart uploads (update metadata and contents at the same time) for improved perfomance & stability - Bug fixes +- Simple option to move/rename files and directories, via `grive -m oldpath newpath` (by Dylan Wulf, wulfd1@tcnj.edu) ### Grive2 v0.4.1 From 94e24d8d553faa363b4df5bec6f4b4aeb1299426 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 11 Nov 2015 12:31:36 +0300 Subject: [PATCH 110/166] Add required package lists into README --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index 9edec32e..9bf77b0e 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,16 @@ There are also some optional dependencies: - libbfd (for backtrace) - binutils (for libiberty, required for compilation in OpenSUSE, Ubuntu, Arch and etc) +On a Debian/Ubuntu/Linux Mint machine just run the following command to install all +these packages: + + sudo apt-get install git cmake build-essential libgcrypt11-dev libyajl-dev \ + libboost-all-dev libcurl4-openssl-dev libexpat1-dev libcppunit-dev binutils-dev + +FreeBSD: + + pkg install git cmake boost-libs yajl libgcrypt pkgconf cppunit libbfd + Grive uses cmake to build. Basic install sequence is mkdir build From 41bf5ba84553ee926c0c1716edbf9b80b2b364b4 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 11 Nov 2015 13:07:36 +0300 Subject: [PATCH 111/166] "Fix" file size at the moment of adding it to ConcatStream (fixes #39) --- libgrive/src/drive2/Syncer2.cc | 3 ++- libgrive/src/util/ConcatStream.cc | 32 +++++++++++++++++++++---------- libgrive/src/util/ConcatStream.hh | 1 + 3 files changed, 25 insertions(+), 11 deletions(-) diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index 4c0f290b..34551812 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -172,10 +172,11 @@ bool Syncer2::Upload( Resource *res ) else { File file( res->Path() ) ; + uint64_t size = file.Size() ; ConcatStream multipart ; StringStream p1( "--file_contents\r\nContent-Type: application/json; charset=utf-8\r\n\r\n" + json_meta + - "\r\n--file_contents\r\nContent-Type: application/octet-stream\r\nContent-Length: " + to_string( file.Size() ) + + "\r\n--file_contents\r\nContent-Type: application/octet-stream\r\nContent-Length: " + to_string( size ) + "\r\n\r\n" ); StringStream p2("\r\n--file_contents--\r\n"); diff --git a/libgrive/src/util/ConcatStream.cc b/libgrive/src/util/ConcatStream.cc index e3c4e1dd..effec729 100644 --- a/libgrive/src/util/ConcatStream.cc +++ b/libgrive/src/util/ConcatStream.cc @@ -17,6 +17,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ +#include #include "ConcatStream.hh" namespace gr { @@ -31,15 +32,22 @@ std::size_t ConcatStream::Read( char *data, std::size_t size ) std::size_t done = 0, l; while ( done < size && m_cur < m_streams.size() ) { - l = m_streams[m_cur]->Read( data+done, size-done ); + l = m_streams[m_cur]->Read( data+done, ( size-done < m_sizes[m_cur]-m_pos ? size-done : m_sizes[m_cur]-m_pos ) ); + done += l; + m_pos += l; if ( !l ) + { + // current stream was truncated in the meantime, pad output with zeros + memset( data+done, 0, m_sizes[m_cur]-m_pos ); + done += m_sizes[m_cur]-m_pos; + m_pos = m_sizes[m_cur]; + } + if ( m_pos >= m_sizes[m_cur] ) { m_cur++; if ( m_cur < m_streams.size() ) m_streams[m_cur]->Seek( 0, 0 ); } - done += l; - m_pos += l; } return done ; } @@ -61,12 +69,9 @@ off_t ConcatStream::Seek( off_t offset, int whence ) m_pos = offset; if ( m_streams.size() ) { - while ( offset > m_streams[m_cur]->Size() ) - { - offset -= m_streams[m_cur]->Size(); + while ( offset > m_sizes[m_cur] ) m_cur++; - } - m_streams[m_cur]->Seek( offset, 0 ); + m_streams[m_cur]->Seek( offset - ( m_cur > 0 ? m_sizes[m_cur-1] : 0 ), 0 ); } return m_pos ; } @@ -85,8 +90,15 @@ void ConcatStream::Append( SeekStream *stream ) { if ( stream ) { - m_streams.push_back( stream ); - m_size += stream->Size(); + off_t size = stream->Size(); + if ( size > 0 ) + { + // "fix" stream size at the moment of adding so further changes of underlying files + // don't affect the total size of resulting stream... + m_size += size; + m_streams.push_back( stream ); + m_sizes.push_back( m_size ); + } } } diff --git a/libgrive/src/util/ConcatStream.hh b/libgrive/src/util/ConcatStream.hh index 82f71236..0ab037e4 100644 --- a/libgrive/src/util/ConcatStream.hh +++ b/libgrive/src/util/ConcatStream.hh @@ -41,6 +41,7 @@ public : private : std::vector m_streams ; + std::vector m_sizes ; off_t m_size, m_pos ; int m_cur ; } ; From cc13b8b34300e4b309a95d983a2cbb363a0d0076 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 9 Dec 2015 01:41:04 +0300 Subject: [PATCH 112/166] Fix finding libbfd on newer Debian --- cmake/Modules/FindBFD.cmake | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/cmake/Modules/FindBFD.cmake b/cmake/Modules/FindBFD.cmake index 785e29b6..35dfd5d6 100644 --- a/cmake/Modules/FindBFD.cmake +++ b/cmake/Modules/FindBFD.cmake @@ -1,5 +1,5 @@ -find_library( DL_LIBRARY NAMES dl PATH /usr/lib /usr/lib64 ) -find_library( BFD_LIBRARY NAMES bfd PATH /usr/lib /usr/lib64 ) +find_library( DL_LIBRARY NAMES dl PATH ${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES} ) +find_library( BFD_LIBRARY NAMES bfd PATH ${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES} ) if ( DL_LIBRARY AND BFD_LIBRARY ) set( BFD_FOUND TRUE ) @@ -9,5 +9,4 @@ if ( BFD_FOUND ) message( STATUS "Found libbfd: ${BFD_LIBRARY}") - endif ( BFD_FOUND ) From 85b35b4fca53b8d44e9d2576bc815079f5ae1468 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 12 Dec 2015 16:44:02 +0300 Subject: [PATCH 113/166] Fix RelPath() for --ignore option to work correctly --- libgrive/src/base/Resource.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 61a82d20..ef3614c9 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -343,7 +343,7 @@ fs::path Resource::RelPath() const assert( m_parent != this ) ; assert( m_parent == 0 || m_parent->IsFolder() ) ; - return m_parent != 0 && !m_parent->IsRoot() ? (m_parent->Path() / m_name) : m_name ; + return m_parent != 0 && !m_parent->IsRoot() ? (m_parent->RelPath() / m_name) : m_name ; } bool Resource::IsInRootTree() const From 7e6e153a3a78731ebe7f69e1f08bd0078ce16403 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 12 Dec 2015 16:59:02 +0300 Subject: [PATCH 114/166] Use regex_search() instead of regex_match(), turn on -f implicitly when changing ignore regexp (should fix #43) --- libgrive/src/base/State.cc | 22 +++++++++++++++++----- libgrive/src/base/State.hh | 3 ++- libgrive/src/util/Config.cc | 5 ++--- 3 files changed, 21 insertions(+), 9 deletions(-) diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index bec157e2..00f17283 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -36,19 +36,29 @@ namespace gr { State::State( const fs::path& filename, const Val& options ) : m_res ( options["path"].Str() ), m_dir ( options["dir"].Str() ), - m_cstamp ( -1 ), - m_ign ( !options["ignore"].Str().empty() ? options["ignore"].Str()+"|^\\.(grive|grive_state|trash)" : "^\\.(grive|grive_state|trash)" ) + m_cstamp ( -1 ) { Read( filename ) ; + bool force = options.Has( "force" ) ? options["force"].Bool() : false ; + + if ( options.Has( "ignore" ) && !m_ign.empty() && options["ignore"].Str() != m_ign ) + { + // also "-f" is implicitly turned on when ignore regexp is changed + // because without it grive would think that previously ignored files are deleted locally + m_ign = options["ignore"].Str(); + force = true; + } + // the "-f" option will make grive always think remote is newer - Val force ; - if ( options.Get("force", force) && force.Bool() ) + if ( force ) { m_last_change = DateTime() ; m_last_sync = DateTime::Now() ; } + m_ign_re = boost::regex( m_ign.empty() ? "^\\.(grive|grive_state|trash)" : ( m_ign+"|^\\.(grive|grive_state|trash)" ) ); + Log( "last server change time: %1%; last sync time: %2%", m_last_change, m_last_sync, log::verbose ) ; } @@ -65,7 +75,7 @@ void State::FromLocal( const fs::path& p ) bool State::IsIgnore( const std::string& filename ) { - return regex_match( filename.c_str(), m_ign ); + return regex_search( filename.c_str(), m_ign_re ); } void State::FromLocal( const fs::path& p, Resource* folder ) @@ -261,6 +271,7 @@ void State::Read( const fs::path& filename ) Val last_change = json.Has( "last_change" ) ? json["last_change"] : json["last_sync"] ; m_last_sync.Assign( last_sync["sec"].Int(), last_sync["nsec"].Int() ) ; m_last_change.Assign( last_change["sec"].Int(), last_change["nsec"].Int() ) ; + m_ign = json.Has( "ignore_regexp" ) ? json["ignore_regexp"].Str() : std::string(); m_cstamp = json["change_stamp"].Int() ; } @@ -283,6 +294,7 @@ void State::Write( const fs::path& filename ) const result.Add( "last_sync", last_sync ) ; result.Add( "last_change", last_change ) ; result.Add( "change_stamp", Val(m_cstamp) ) ; + result.Add( "ignore_regexp", Val(m_ign) ) ; std::ofstream fs( filename.string().c_str() ) ; fs << result ; diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 7822be97..8df66d40 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -79,7 +79,8 @@ private : DateTime m_last_change ; int m_cstamp ; std::string m_dir ; - boost::regex m_ign ; + std::string m_ign ; + boost::regex m_ign_re ; std::vector m_unresolved ; } ; diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index 69ac344c..b8483c82 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -46,9 +46,8 @@ Config::Config( const po::variables_map& vm ) m_cmd.Add( "dir", Val(vm.count("dir") > 0 ? vm["dir"].as() : "" ) ) ; - m_cmd.Add( "ignore", Val(vm.count("ignore") > 0 - ? vm["ignore"].as() - : "" ) ) ; + if ( vm.count("ignore") > 0 ) + m_cmd.Add( "ignore", Val( vm["ignore"].as() ) ); m_path = GetPath( fs::path(m_cmd["path"].Str()) ) ; m_file = Read( ) ; From 9c43ad9cc94ae5d0761538fa88488321e95996e8 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 12 Dec 2015 17:16:05 +0300 Subject: [PATCH 115/166] Internally convert -s option to ignore regexp. Fixes #49, should fix #42. --- libgrive/src/base/State.cc | 24 +++++++++++++++--------- libgrive/src/base/State.hh | 1 - 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 00f17283..1f79e3dc 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -35,7 +35,6 @@ namespace gr { State::State( const fs::path& filename, const Val& options ) : m_res ( options["path"].Str() ), - m_dir ( options["dir"].Str() ), m_cstamp ( -1 ) { Read( filename ) ; @@ -49,6 +48,20 @@ State::State( const fs::path& filename, const Val& options ) : m_ign = options["ignore"].Str(); force = true; } + else if ( options.Has( "dir" ) ) + { + const boost::regex trim_path( "^/+|/+$" ); + std::string m_dir = regex_replace( options["dir"].Str(), trim_path, "" ); + if ( !m_dir.empty() ) + { + // "-s" is internally converted to an ignore regexp + const boost::regex esc( "[.^$|()\\[\\]{}*+?\\\\]" ); + std::string ign = "^(?!"+regex_replace( m_dir, esc, "\\\\&", boost::format_sed )+"(/|$))"; + if ( !m_ign.empty() && ign != m_ign ) + force = true; + m_ign = ign; + } + } // the "-f" option will make grive always think remote is newer if ( force ) @@ -95,10 +108,6 @@ void State::FromLocal( const fs::path& p, Resource* folder ) if ( IsIgnore( path ) ) Log( "file %1% is ignored by grive", path, log::verbose ) ; - // check if it is ignored - else if ( folder == m_res.Root() && m_dir != "" && fname != m_dir ) - Log( "%1% %2% is ignored", st.type() == fs::directory_file ? "folder" : "file", fname, log::verbose ); - // check for broken symblic links else if ( st.type() == fs::file_not_found ) Log( "file %1% doesn't exist (broken link?), ignored", i->path(), log::verbose ) ; @@ -129,11 +138,8 @@ void State::FromRemote( const Entry& e ) std::string fn = e.Filename() ; std::string k = e.IsDir() ? "folder" : "file"; - if ( e.ParentHref() == m_res.Root()->SelfHref() && m_dir != "" && e.Name() != m_dir ) - Log( "%1% %2% is ignored", k, e.Name(), log::verbose ); - // common checkings - else if ( !e.IsDir() && (fn.empty() || e.ContentSrc().empty()) ) + if ( !e.IsDir() && (fn.empty() || e.ContentSrc().empty()) ) Log( "%1% \"%2%\" is a google document, ignored", k, e.Name(), log::verbose ) ; else if ( fn.find('/') != fn.npos ) diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 8df66d40..d9728021 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -78,7 +78,6 @@ private : DateTime m_last_sync ; DateTime m_last_change ; int m_cstamp ; - std::string m_dir ; std::string m_ign ; boost::regex m_ign_re ; From c0fadcffe3b16275d0b56f8e12448fbec1b9fe4e Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 12 Dec 2015 17:22:17 +0300 Subject: [PATCH 116/166] State that -s and --ignore are remembered for next runs of Grive in --help --- README.md | 2 +- grive/src/main.cc | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 9bf77b0e..c7e67a67 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Grive2 0.4.2-dev -10 Oct 2015, Vitaliy Filippov +12 Dec 2015, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 diff --git a/grive/src/main.cc b/grive/src/main.cc index 0472b1a5..347f3235 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -109,8 +109,8 @@ int Main( int argc, char **argv ) ( "help,h", "Produce help message" ) ( "version,v", "Display Grive version" ) ( "auth,a", "Request authorization token" ) - ( "path,p", po::value(), "Path to sync") - ( "dir,s", po::value(), "Subdirectory to sync") + ( "path,p", po::value(), "Root directory to sync") + ( "dir,s", po::value(), "Single subdirectory to sync (remembered for next runs)") ( "verbose,V", "Verbose mode. Enable more messages than normal.") ( "log-http", po::value(), "Log all HTTP responses in this file for debugging.") ( "new-rev", "Create new revisions in server for updated files.") @@ -120,7 +120,7 @@ int Main( int argc, char **argv ) "instead of uploading it." ) ( "dry-run", "Only detect which files need to be uploaded/downloaded, " "without actually performing them." ) - ( "ignore", po::value(), "Ignore files relative paths of which match this Perl RegExp." ) + ( "ignore", po::value(), "Perl RegExp to ignore files (matched against relative paths, remembered for next runs)." ) ( "move,m", po::value >()->multitoken(), "Syncs, then moves a file (first argument) to new location (second argument) without reuploading or redownloading." ) ; From ff9d7bcd148be60b0a467b9b1f655af465ebca82 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 13 Dec 2015 01:53:35 +0300 Subject: [PATCH 117/166] Warn, but not die, on filesystem exceptions during sync. Fixes #44 --- libgrive/src/base/Resource.cc | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index ef3614c9..78605167 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -28,8 +28,12 @@ #include "util/OS.hh" #include "util/File.hh" +#include +#include #include +#include + #include // for debugging @@ -369,7 +373,21 @@ void Resource::Sync( Syncer *syncer, DateTime& sync_time, const Val& options ) assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced - SyncSelf( syncer, options ) ; + try + { + SyncSelf( syncer, options ) ; + } + catch ( File::Error &e ) + { + int *en = boost::get_error_info< boost::errinfo_errno > ( e ) ; + Log( "Error syncing %1%: %2%", Path(), en ? strerror( *en ) : "", log::error ); + return; + } + catch ( boost::filesystem::filesystem_error &e ) + { + Log( "Error syncing %1%: %2%", Path(), e.what(), log::error ); + return; + } // we want the server sync time, so we will take the server time of the last file uploaded to store as the sync time // m_mtime is updated to server modified time when the file is uploaded From 5fb3c187fef4b257083a6e954d037f157fa9730c Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Mon, 28 Dec 2015 02:00:47 +0300 Subject: [PATCH 118/166] Oops this was a fail. ignore option was ignored if it was previously empty in grive_state. Fixes #50 --- libgrive/src/base/State.cc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 1f79e3dc..fb64a47a 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -41,12 +41,13 @@ State::State( const fs::path& filename, const Val& options ) : bool force = options.Has( "force" ) ? options["force"].Bool() : false ; - if ( options.Has( "ignore" ) && !m_ign.empty() && options["ignore"].Str() != m_ign ) + if ( options.Has( "ignore" ) && options["ignore"].Str() != m_ign ) { // also "-f" is implicitly turned on when ignore regexp is changed // because without it grive would think that previously ignored files are deleted locally + if ( !m_ign.empty() ) + force = true; m_ign = options["ignore"].Str(); - force = true; } else if ( options.Has( "dir" ) ) { From d2a6105603e8e4109787170faddd7868000c2d45 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 2 Jan 2016 15:29:55 +0300 Subject: [PATCH 119/166] Release 0.4.2 --- CMakeLists.txt | 2 +- README.md | 9 ++++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index c810ff9f..d0ba0d54 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! -set( GRIVE_VERSION "0.4.2-pre" ) +set( GRIVE_VERSION "0.4.2" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) diff --git a/README.md b/README.md index c7e67a67..ca6e53e8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# Grive2 0.4.2-dev +# Grive2 0.4.2 -12 Dec 2015, Vitaliy Filippov +28 Dec 2015, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 @@ -70,7 +70,7 @@ Enjoy! ## Version History -### Grive2 v0.4.2 (unreleased) +### Grive2 v0.4.2 - Option to exclude files by perl regexp - Reimplemented HTTP response logging for debug purposes @@ -78,6 +78,9 @@ Enjoy! - Bug fixes - Simple option to move/rename files and directories, via `grive -m oldpath newpath` (by Dylan Wulf, wulfd1@tcnj.edu) +Known issues: +- force option does not work as documented #51 + ### Grive2 v0.4.1 - Bug fixes From 5381919e5b469ac0aa8b9d6d874441073c199c23 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 1 Jan 2016 18:15:34 +0300 Subject: [PATCH 120/166] Enable warnings and fix them --- libgrive/CMakeLists.txt | 2 ++ libgrive/src/base/Resource.cc | 12 ++++----- libgrive/src/base/Resource.hh | 1 + libgrive/src/base/State.cc | 42 +++++++++++++++---------------- libgrive/src/drive2/Syncer2.cc | 7 +++--- libgrive/src/protocol/OAuth2.cc | 2 +- libgrive/src/util/ConcatStream.cc | 8 +++--- libgrive/src/util/ConcatStream.hh | 6 ++--- libgrive/src/util/StringStream.cc | 2 +- 9 files changed, 41 insertions(+), 41 deletions(-) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index f3aa7757..90ef451d 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -14,6 +14,8 @@ find_package(ZLIB) find_package(PkgConfig) pkg_check_modules(YAJL REQUIRED yajl) +add_definitions(-Wall) + # additional headers if build unit tests IF ( CPPUNIT_FOUND ) set( OPT_INCS ${CPPUNIT_INCLUDE_DIR} ) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 78605167..9b75292e 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -47,18 +47,18 @@ Resource::Resource( const fs::path& root_folder ) : m_kind ( "folder" ), m_id ( "folder:root" ), m_href ( "root" ), - m_parent ( 0 ), - m_state ( sync ), - m_is_editable( true ) + m_is_editable( true ), + m_parent ( NULL ), + m_state ( sync ) { } Resource::Resource( const std::string& name, const std::string& kind ) : m_name ( name ), m_kind ( kind ), - m_parent ( 0 ), - m_state ( unknown ), - m_is_editable( true ) + m_is_editable( true ), + m_parent ( NULL ), + m_state ( unknown ) { } diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index cb712c62..bc342a71 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -152,6 +152,7 @@ private : std::vector m_child ; State m_state ; + Val* m_json ; } ; } // end of namespace gr::v1 diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index fb64a47a..c3f7cc2b 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -342,51 +342,49 @@ void State::ChangeStamp( long cstamp ) bool State::Move( Syncer* syncer, fs::path old_p, fs::path new_p, fs::path grive_root ) { - //Convert paths to canonical representations - //Also seems to remove trailing / at the end of directory paths + // Convert paths to canonical representations + // Also seems to remove trailing / at the end of directory paths old_p = fs::canonical( old_p ); grive_root = fs::canonical( grive_root ); - //new_p is a little special because fs::canonical() requires that the path exists - if ( new_p.string()[ new_p.string().size() - 1 ] == '/') //If new_p ends with a /, remove it + // new_p is a little special because fs::canonical() requires that the path exists + if ( new_p.string()[ new_p.string().size() - 1 ] == '/') // If new_p ends with a /, remove it new_p = new_p.parent_path(); new_p = fs::canonical( new_p.parent_path() ) / new_p.filename(); - //Fails if source file doesn't exist, or if destination file already - //exists and is not a directory, or if the source and destination are exactly the same - if ( (fs::exists(new_p) && !fs::is_directory(new_p) ) || !fs::exists(old_p) || fs::equivalent( old_p, new_p ) ) + // Fails if source file doesn't exist, or if destination file already + // exists and is not a directory, or if the source and destination are exactly the same + if ( (fs::exists(new_p) && !fs::is_directory(new_p)) || !fs::exists(old_p) || fs::equivalent( old_p, new_p ) ) return false; - //If new path is an existing directory, move the file into the directory - //instead of trying to rename it - if ( fs::is_directory(new_p) ){ + // If new path is an existing directory, move the file into the directory + // instead of trying to rename it + if ( fs::is_directory( new_p ) ) new_p = new_p / old_p.filename(); - } - //Get the paths relative to grive root. - //Just finds the substring from the end of the grive_root to the end of the path - //+1s are to exclude slash at beginning of relative path - int start = grive_root.string().size() + 1; - int nLen = new_p.string().size() - (grive_root.string().size() + 1); - int oLen = old_p.string().size() - (grive_root.string().size() + 1); - if ( start + nLen != new_p.string().size() || start + oLen != old_p.string().size() ) + // Get the paths relative to grive root. + // Just finds the substring from the end of the grive_root to the end of the path + // +1s are to exclude slash at beginning of relative path + std::string root( grive_root.string() + "/" ); + if ( new_p.string().substr( 0, root.length() ).compare( root ) != 0 || + old_p.string().substr( 0, root.length() ).compare( root ) != 0 ) return false; - fs::path new_p_rootrel( new_p.string().substr( start, nLen ) ); - fs::path old_p_rootrel( old_p.string().substr( start, oLen ) ); + fs::path new_p_rootrel( new_p.string().substr( root.length() ) ); + fs::path old_p_rootrel( old_p.string().substr( root.length() ) ); //Get resources Resource* res = m_res.Root(); Resource* newParentRes = m_res.Root(); for ( fs::path::iterator it = old_p_rootrel.begin(); it != old_p_rootrel.end(); ++it ) { - if ( *it != "." && *it != ".." && res != 0 ) + if ( *it != "." && *it != ".." && res ) res = res->FindChild(it->string()); if ( *it == ".." ) res = res->Parent(); } for ( fs::path::iterator it = new_p_rootrel.begin(); it != new_p_rootrel.end(); ++it ) { - if ( *it != "." && *it != ".." && *it != new_p.filename() && newParentRes != 0 ) + if ( *it != "." && *it != ".." && *it != new_p.filename() && newParentRes ) newParentRes = newParentRes->FindChild(it->string()); if ( *it == "..") res = res->Parent(); diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index 34551812..5151279f 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -121,11 +121,10 @@ bool Syncer2::Move( Resource* res, Resource* newParentRes, std::string newFilena http::Header hdr2 ; hdr2.Add( "Content-Type: application/json" ); http::ValResponse vrsp ; - long http_code = 0; //Don't change modified date because we're only moving - http_code = m_http->Put( feeds::files + "/" + res->ResourceID() + "?modifiedDateBehavior=noChange" + addRemoveParents, json_meta, &vrsp, hdr2 ) ; + long http_code = m_http->Put( feeds::files + "/" + res->ResourceID() + "?modifiedDateBehavior=noChange" + addRemoveParents, json_meta, &vrsp, hdr2 ) ; valr = vrsp.Response(); - assert( !( valr["id"].Str().empty() ) ); + assert( http_code == 200 && !( valr["id"].Str().empty() ) ); } return true; } @@ -167,7 +166,7 @@ bool Syncer2::Upload( Resource *res ) else http_code = m_http->Put( feeds::files + "/" + res->ResourceID(), json_meta, &vrsp, hdr2 ) ; valr = vrsp.Response(); - assert( !( valr["id"].Str().empty() ) ); + assert( http_code == 200 && !( valr["id"].Str().empty() ) ); } else { diff --git a/libgrive/src/protocol/OAuth2.cc b/libgrive/src/protocol/OAuth2.cc index cd3dd6ac..73ae0886 100644 --- a/libgrive/src/protocol/OAuth2.cc +++ b/libgrive/src/protocol/OAuth2.cc @@ -37,8 +37,8 @@ OAuth2::OAuth2( const std::string& refresh_code, const std::string& client_id, const std::string& client_secret ) : - m_agent( agent ), m_refresh( refresh_code ), + m_agent( agent ), m_client_id( client_id ), m_client_secret( client_secret ) { diff --git a/libgrive/src/util/ConcatStream.cc b/libgrive/src/util/ConcatStream.cc index effec729..d93df5e0 100644 --- a/libgrive/src/util/ConcatStream.cc +++ b/libgrive/src/util/ConcatStream.cc @@ -23,7 +23,7 @@ namespace gr { ConcatStream::ConcatStream() : - m_cur( 0 ), m_size( 0 ), m_pos( 0 ) + m_size( 0 ), m_pos( 0 ), m_cur( 0 ) { } @@ -63,13 +63,13 @@ off_t ConcatStream::Seek( off_t offset, int whence ) offset += m_pos; else if ( whence == 2 ) offset += Size(); - if ( offset > Size() ) + if ( (u64_t)offset > Size() ) offset = Size(); m_cur = 0; m_pos = offset; if ( m_streams.size() ) { - while ( offset > m_sizes[m_cur] ) + while ( (u64_t)offset > m_sizes[m_cur] ) m_cur++; m_streams[m_cur]->Seek( offset - ( m_cur > 0 ? m_sizes[m_cur-1] : 0 ), 0 ); } @@ -90,7 +90,7 @@ void ConcatStream::Append( SeekStream *stream ) { if ( stream ) { - off_t size = stream->Size(); + u64_t size = stream->Size(); if ( size > 0 ) { // "fix" stream size at the moment of adding so further changes of underlying files diff --git a/libgrive/src/util/ConcatStream.hh b/libgrive/src/util/ConcatStream.hh index 0ab037e4..0966f060 100644 --- a/libgrive/src/util/ConcatStream.hh +++ b/libgrive/src/util/ConcatStream.hh @@ -41,9 +41,9 @@ public : private : std::vector m_streams ; - std::vector m_sizes ; - off_t m_size, m_pos ; - int m_cur ; + std::vector m_sizes ; + u64_t m_size, m_pos ; + size_t m_cur ; } ; } // end of namespace diff --git a/libgrive/src/util/StringStream.cc b/libgrive/src/util/StringStream.cc index a128f746..447f8928 100644 --- a/libgrive/src/util/StringStream.cc +++ b/libgrive/src/util/StringStream.cc @@ -51,7 +51,7 @@ off_t StringStream::Seek( off_t offset, int whence ) offset += m_pos; else if ( whence == 2 ) offset += Size(); - if ( offset > Size() ) + if ( (u64_t)offset > Size() ) offset = Size(); m_pos = (size_t)offset; return m_pos; From 4edff0a8169a7a57829e44dc5ff5ced15b363933 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 1 Jan 2016 02:27:21 +0300 Subject: [PATCH 121/166] Cache file checksums and modification times in local index (in JSON in .grive_state) --- libgrive/src/base/Drive.cc | 4 +- libgrive/src/base/Resource.cc | 107 +++++++++++++++++++++++------ libgrive/src/base/Resource.hh | 8 ++- libgrive/src/base/State.cc | 66 +++++++++--------- libgrive/src/base/State.hh | 8 +-- libgrive/src/base/Syncer.cc | 9 +-- libgrive/src/base/Syncer.hh | 1 - libgrive/src/drive/Syncer1.cc | 2 +- libgrive/src/drive2/Syncer2.cc | 2 +- libgrive/src/json/Val.cc | 50 ++++++++++++++ libgrive/src/json/Val.hh | 23 ++++--- libgrive/src/util/OS.cc | 17 +++-- libgrive/src/util/OS.hh | 4 +- libgrive/test/base/ResourceTest.cc | 4 +- 14 files changed, 214 insertions(+), 91 deletions(-) diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index 63128a17..9190111b 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -62,11 +62,11 @@ void Drive::FromRemote( const Entry& entry ) // so these checkings are done in normal entries only Resource *parent = m_state.FindByHref( entry.ParentHref() ) ; - if ( parent != 0 && !parent->IsFolder() ) + if ( parent && !parent->IsFolder() ) Log( "warning: entry %1% has parent %2% which is not a folder, ignored", entry.Title(), parent->Name(), log::verbose ) ; - else if ( parent == 0 || !parent->IsInRootTree() ) + else if ( !parent || !parent->IsInRootTree() ) Log( "file \"%1%\" parent doesn't exist, ignored", entry.Title(), log::verbose ) ; else diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 9b75292e..eb9c74c8 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -48,8 +48,9 @@ Resource::Resource( const fs::path& root_folder ) : m_id ( "folder:root" ), m_href ( "root" ), m_is_editable( true ), - m_parent ( NULL ), - m_state ( sync ) + m_parent ( 0 ), + m_state ( sync ), + m_json ( NULL ) { } @@ -57,8 +58,9 @@ Resource::Resource( const std::string& name, const std::string& kind ) : m_name ( name ), m_kind ( kind ), m_is_editable( true ), - m_parent ( NULL ), - m_state ( unknown ) + m_parent ( 0 ), + m_state ( unknown ), + m_json ( NULL ) { } @@ -137,10 +139,8 @@ void Resource::FromRemote( const Entry& remote, const DateTime& last_change ) assert( m_state != unknown ) ; if ( m_state == remote_new || m_state == remote_changed ) - { - m_md5 = remote.MD5() ; - m_mtime = remote.MTime() ; - } + m_md5 = remote.MD5() ; + m_mtime = remote.MTime() ; } void Resource::AssignIDs( const Entry& remote ) @@ -233,15 +233,40 @@ void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_change /// Update the resource with the attributes of local file or directory. This /// function will propulate the fields in m_entry. -void Resource::FromLocal( const DateTime& last_sync ) +void Resource::FromLocal( const DateTime& last_sync, Val& state ) { - fs::path path = Path() ; - //assert( fs::exists( path ) ) ; + assert( !m_json ); + m_json = &state; // root folder is always in sync if ( !IsRoot() ) { - m_mtime = os::FileCTime( path ) ; + fs::path path = Path() ; + bool is_dir; + os::Stat( path, &m_ctime, &m_size, &is_dir ) ; + + m_name = path.filename().string() ; + if ( !is_dir ) + { + m_kind = "file"; + if ( state.Has( "ctime" ) && state.Has( "md5" ) && (u64_t) m_ctime.Sec() == state["ctime"].U64() ) + m_md5 = state["md5"]; + else + { + m_md5 = crypt::MD5::Get( path ); + state.Set( "md5", Val( m_md5 ) ); + state.Set( "ctime", Val( m_ctime.Sec() ) ); + } + if ( state.Has( "srv_time" ) && m_mtime != DateTime() ) + m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; + } + else + { + m_kind = "folder"; + state.Del( "md5" ); + state.Del( "ctime" ); + state.Del( "srv_time" ); + } // follow parent recursively if ( m_parent->m_state == local_new || m_parent->m_state == local_deleted ) @@ -251,11 +276,7 @@ void Resource::FromLocal( const DateTime& last_sync ) // remote_deleted first, it will be updated to sync/remote_changed // in FromRemote() else - m_state = ( m_mtime > last_sync ? local_new : remote_deleted ) ; - - m_name = path.filename().string() ; - m_kind = IsFolder() ? "folder" : "file" ; - m_md5 = IsFolder() ? "" : crypt::MD5::Get( path ) ; + m_state = ( m_ctime > last_sync ? local_new : remote_deleted ) ; } assert( m_state != unknown ) ; @@ -286,7 +307,7 @@ std::string Resource::Kind() const return m_kind ; } -DateTime Resource::MTime() const +DateTime Resource::ServerTime() const { return m_mtime ; } @@ -395,8 +416,26 @@ void Resource::Sync( Syncer *syncer, DateTime& sync_time, const Val& options ) // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) + { std::for_each( m_child.begin(), m_child.end(), boost::bind( &Resource::Sync, _1, syncer, boost::ref(sync_time), options ) ) ; + if ( IsFolder() ) + { + // delete state of removed files + Val& tree = (*m_json)["tree"]; + Val::Object leftover = tree.AsObject(); + for( iterator i = m_child.begin(); i != m_child.end(); i++ ) + { + Resource *r = *i; + if ( r->m_state != local_deleted && r->m_state != remote_deleted ) + leftover.erase( r->Name() ); + else + r->m_json = NULL; + } + for( Val::Object::iterator i = leftover.begin(); i != leftover.end(); i++ ) + tree.Del( i->first ); + } + } } void Resource::SyncSelf( Syncer* syncer, const Val& options ) @@ -438,7 +477,7 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) fs::create_directories( path ) ; else syncer->Download( this, path ) ; - + SetIndex() ; m_state = sync ; } break ; @@ -449,6 +488,7 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) if ( syncer ) { syncer->Download( this, path ) ; + SetIndex() ; m_state = sync ; } break ; @@ -462,7 +502,7 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) case sync : Log( "sync %1% already in sync", path, log::verbose ) ; break ; - + // shouldn't go here case unknown : assert( false ) ; @@ -471,6 +511,17 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) default : break ; } + + if ( m_state != local_deleted && m_state != remote_deleted && !IsFolder() ) + { + // Update server time of this file + m_json->Set( "srv_time", Val( m_mtime.Sec() ) ); + } +} + +void Resource::SetServerTime( const DateTime& time ) +{ + m_mtime = time ; } /// this function doesn't really remove the local file. it renames it. @@ -478,7 +529,7 @@ void Resource::DeleteLocal() { static const boost::format trash_file( "%1%-%2%" ) ; - assert( m_parent != 0 ) ; + assert( m_parent != NULL ) ; Resource* p = m_parent; fs::path destdir; while ( !p->IsRoot() ) @@ -503,6 +554,20 @@ void Resource::DeleteLocal() } } +void Resource::SetIndex() +{ + assert( m_parent->m_json != NULL ); + if ( !m_json ) + m_json = &((*m_parent->m_json)["tree"]).Item( Name() ); + bool is_dir; + os::Stat( Path(), &m_ctime, &m_size, &is_dir ); + if ( !is_dir ) + { + m_json->Set( "ctime", Val( m_ctime.Sec() ) ); + m_json->Set( "md5", Val( m_md5 ) ); + } +} + Resource::iterator Resource::begin() const { return m_child.begin() ; diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index bc342a71..99b93154 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -87,7 +87,7 @@ public : std::string Name() const ; std::string Kind() const ; - DateTime MTime() const ; + DateTime ServerTime() const ; std::string SelfHref() const ; std::string ContentSrc() const ; std::string ETag() const ; @@ -107,9 +107,10 @@ public : std::string MD5() const ; void FromRemote( const Entry& remote, const DateTime& last_change ) ; - void FromLocal( const DateTime& last_sync ) ; + void FromLocal( const DateTime& last_sync, Val& state ) ; void Sync( Syncer* syncer, DateTime& sync_time, const Val& options ) ; + void SetServerTime( const DateTime& time ) ; // children access iterator begin() const ; @@ -132,6 +133,7 @@ private : void FromRemoteFile( const Entry& remote, const DateTime& last_change ) ; void DeleteLocal() ; + void SetIndex() ; void SyncSelf( Syncer* syncer, const Val& options ) ; @@ -140,6 +142,8 @@ private : std::string m_kind ; std::string m_md5 ; DateTime m_mtime ; + DateTime m_ctime ; + off_t m_size ; std::string m_id ; std::string m_href ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index c3f7cc2b..0cfa5f83 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -26,7 +26,6 @@ #include "util/Crypt.hh" #include "util/File.hh" #include "util/log/Log.hh" -#include "json/Val.hh" #include "json/JsonParser.hh" #include @@ -84,7 +83,10 @@ State::~State() /// of local directory. void State::FromLocal( const fs::path& p ) { - FromLocal( p, m_res.Root() ) ; + if ( !m_st.Has( "tree" ) ) + m_st.Add( "tree", Val() ); + m_res.Root()->FromLocal( m_last_sync, m_st ) ; + FromLocal( p, m_res.Root(), m_st["tree"] ) ; } bool State::IsIgnore( const std::string& filename ) @@ -92,44 +94,39 @@ bool State::IsIgnore( const std::string& filename ) return regex_search( filename.c_str(), m_ign_re ); } -void State::FromLocal( const fs::path& p, Resource* folder ) +void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) { assert( folder != 0 ) ; assert( folder->IsFolder() ) ; - // sync the folder itself - folder->FromLocal( m_last_sync ) ; - for ( fs::directory_iterator i( p ) ; i != fs::directory_iterator() ; ++i ) { std::string fname = i->path().filename().string() ; - fs::file_status st = fs::status(i->path()); - std::string path = folder->IsRoot() ? fname : ( folder->RelPath() / fname ).string(); + if ( IsIgnore( path ) ) Log( "file %1% is ignored by grive", path, log::verbose ) ; - - // check for broken symblic links - else if ( st.type() == fs::file_not_found ) - Log( "file %1% doesn't exist (broken link?), ignored", i->path(), log::verbose ) ; - else { - bool is_dir = st.type() == fs::directory_file; // if the Resource object of the child already exists, it should // have been so no need to do anything here Resource *c = folder->FindChild( fname ) ; - if ( c == 0 ) + if ( !c ) { - c = new Resource( fname, is_dir ? "folder" : "file" ) ; + c = new Resource( fname, "" ) ; folder->AddChild( c ) ; m_res.Insert( c ) ; } - - c->FromLocal( m_last_sync ) ; - - if ( is_dir ) - FromLocal( *i, c ) ; + if ( !tree.Has( fname ) ) + tree.Add( fname, Val() ); + Val& rec = tree[fname]; + c->FromLocal( m_last_sync, rec ) ; + if ( c->IsFolder() ) + { + if ( !rec.Has("tree") ) + rec.Add( "tree", Val() ); + FromLocal( *i, c, rec["tree"] ) ; + } } } } @@ -140,7 +137,7 @@ void State::FromRemote( const Entry& e ) std::string k = e.IsDir() ? "folder" : "file"; // common checkings - if ( !e.IsDir() && (fn.empty() || e.ContentSrc().empty()) ) + if ( !e.IsDir() && ( fn.empty() || e.ContentSrc().empty() ) ) Log( "%1% \"%2%\" is a google document, ignored", k, e.Name(), log::verbose ) ; else if ( fn.find('/') != fn.npos ) @@ -225,7 +222,7 @@ bool State::Update( const Entry& e ) // see if the entry already exist in local std::string name = e.Name() ; Resource *child = parent->FindChild( name ) ; - if ( child != 0 ) + if ( child ) { // since we are updating the ID and Href, we need to remove it and re-add it. m_res.Update( child, e, m_last_change ) ; @@ -273,21 +270,21 @@ void State::Read( const fs::path& filename ) { File file( filename ) ; - Val json = ParseJson( file ); - Val last_sync = json["last_sync"] ; - Val last_change = json.Has( "last_change" ) ? json["last_change"] : json["last_sync"] ; + m_st = ParseJson( file ); + Val last_sync = m_st["last_sync"] ; + Val last_change = m_st.Has( "last_change" ) ? m_st["last_change"] : m_st["last_sync"] ; m_last_sync.Assign( last_sync["sec"].Int(), last_sync["nsec"].Int() ) ; m_last_change.Assign( last_change["sec"].Int(), last_change["nsec"].Int() ) ; - m_ign = json.Has( "ignore_regexp" ) ? json["ignore_regexp"].Str() : std::string(); + m_ign = m_st.Has( "ignore_regexp" ) ? m_st["ignore_regexp"].Str() : std::string(); - m_cstamp = json["change_stamp"].Int() ; + m_cstamp = m_st["change_stamp"].Int() ; } catch ( Exception& ) { } } -void State::Write( const fs::path& filename ) const +void State::Write( const fs::path& filename ) { Val last_sync ; last_sync.Add( "sec", Val( (int)m_last_sync.Sec() ) ); @@ -297,14 +294,13 @@ void State::Write( const fs::path& filename ) const last_change.Add( "sec", Val( (int)m_last_change.Sec() ) ); last_change.Add( "nsec", Val( (unsigned)m_last_change.NanoSec() ) ); - Val result ; - result.Add( "last_sync", last_sync ) ; - result.Add( "last_change", last_change ) ; - result.Add( "change_stamp", Val(m_cstamp) ) ; - result.Add( "ignore_regexp", Val(m_ign) ) ; + m_st.Set( "last_sync", last_sync ) ; + m_st.Set( "last_change", last_change ) ; + m_st.Set( "change_stamp", Val( m_cstamp ) ) ; + m_st.Set( "ignore_regexp", Val( m_ign ) ) ; std::ofstream fs( filename.string().c_str() ) ; - fs << result ; + fs << m_st ; } void State::Sync( Syncer *syncer, const Val& options ) diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index d9728021..353031d6 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -23,14 +23,13 @@ #include "util/DateTime.hh" #include "util/FileSystem.hh" +#include "json/Val.hh" #include #include namespace gr { -class Val ; - class Entry ; class Syncer ; @@ -51,7 +50,7 @@ public : void ResolveEntry() ; void Read( const fs::path& filename ) ; - void Write( const fs::path& filename ) const ; + void Write( const fs::path& filename ) ; Resource* FindByHref( const std::string& href ) ; Resource* FindByID( const std::string& id ) ; @@ -66,7 +65,7 @@ public : bool Move( Syncer* syncer, fs::path old_p, fs::path new_p, fs::path grive_root ); private : - void FromLocal( const fs::path& p, Resource *folder ) ; + void FromLocal( const fs::path& p, Resource *folder, Val& tree ) ; void FromChange( const Entry& e ) ; bool Update( const Entry& e ) ; std::size_t TryResolveEntry() ; @@ -80,6 +79,7 @@ private : int m_cstamp ; std::string m_ign ; boost::regex m_ign_re ; + Val m_st ; std::vector m_unresolved ; } ; diff --git a/libgrive/src/base/Syncer.cc b/libgrive/src/base/Syncer.cc index aa3b031f..2cc312c0 100644 --- a/libgrive/src/base/Syncer.cc +++ b/libgrive/src/base/Syncer.cc @@ -44,8 +44,8 @@ void Syncer::Download( Resource *res, const fs::path& file ) long r = m_http->Get( res->ContentSrc(), &dl, http::Header() ) ; if ( r <= 400 ) { - if ( res->MTime() != DateTime() ) - os::SetFileTime( file, res->MTime() ) ; + if ( res->ServerTime() != DateTime() ) + os::SetFileTime( file, res->ServerTime() ) ; else Log( "encountered zero date time after downloading %1%", file, log::warning ) ; } @@ -56,9 +56,4 @@ void Syncer::AssignIDs( Resource *res, const Entry& remote ) res->AssignIDs( remote ); } -void Syncer::AssignMTime( Resource *res, const DateTime& mtime ) -{ - res->m_mtime = mtime; -} - } // end of namespace gr diff --git a/libgrive/src/base/Syncer.hh b/libgrive/src/base/Syncer.hh index 19adacb5..29de4b99 100644 --- a/libgrive/src/base/Syncer.hh +++ b/libgrive/src/base/Syncer.hh @@ -65,7 +65,6 @@ protected: http::Agent *m_http; void AssignIDs( Resource *res, const Entry& remote ); - void AssignMTime( Resource *res, const DateTime& mtime ); } ; diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index 9ea8a2d4..8eaa9bdd 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -232,7 +232,7 @@ bool Syncer1::Upload( Resource *res, Log( "upload succeeded on retry", log::warning ); Entry1 responseEntry = Entry1( xml.Response() ); AssignIDs( res, responseEntry ) ; - AssignMTime( res, responseEntry.MTime() ); + res->SetServerTime( responseEntry.MTime() ); break; } diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index 5151279f..9d77fa50 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -201,7 +201,7 @@ bool Syncer2::Upload( Resource *res ) Entry2 responseEntry = Entry2( valr ) ; AssignIDs( res, responseEntry ) ; - AssignMTime( res, responseEntry.MTime() ) ; + res->SetServerTime( responseEntry.MTime() ); return true ; } diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index 1debddc2..0d8ca656 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -91,6 +91,18 @@ const Val& Val::operator[]( const std::string& key ) const throw ; } +Val& Val::operator[]( const std::string& key ) +{ + Object& obj = As() ; + Object::iterator i = obj.find(key) ; + if ( i != obj.end() ) + return i->second ; + + // shut off compiler warning + BOOST_THROW_EXCEPTION(Error() << NoKey_(key)) ; + throw ; +} + const Val& Val::operator[]( std::size_t index ) const { const Array& ar = As() ; @@ -119,6 +131,13 @@ int Val::Int() const return static_cast(As()) ; } +unsigned long long Val::U64() const +{ + if ( Type() == string_type ) + return strtoull( As().c_str(), NULL, 10 ); + return static_cast(As()) ; +} + double Val::Double() const { if ( Type() == string_type ) @@ -136,17 +155,38 @@ const Val::Array& Val::AsArray() const return As() ; } +Val::Array& Val::AsArray() +{ + return As() ; +} + const Val::Object& Val::AsObject() const { return As() ; } +Val::Object& Val::AsObject() +{ + return As() ; +} + bool Val::Has( const std::string& key ) const { const Object& obj = As() ; return obj.find(key) != obj.end() ; } +bool Val::Del( const std::string& key ) +{ + Object& obj = As() ; + return obj.erase(key) > 0 ; +} + +Val& Val::Item( const std::string& key ) +{ + return As()[key]; +} + bool Val::Get( const std::string& key, Val& val ) const { const Object& obj = As() ; @@ -165,6 +205,16 @@ void Val::Add( const std::string& key, const Val& value ) As().insert( std::make_pair(key, value) ) ; } +void Val::Set( const std::string& key, const Val& value ) +{ + Object& obj = As(); + Object::iterator i = obj.find(key); + if (i == obj.end()) + obj.insert(std::make_pair(key, value)); + else + i->second = value; +} + void Val::Add( const Val& json ) { As().push_back( json ) ; diff --git a/libgrive/src/json/Val.hh b/libgrive/src/json/Val.hh index e5673b25..dfc2d331 100644 --- a/libgrive/src/json/Val.hh +++ b/libgrive/src/json/Val.hh @@ -94,24 +94,29 @@ public : TypeEnum Type() const ; - const Val& operator[]( const std::string& key ) const ; - const Val& operator[]( std::size_t index ) const ; - // shortcuts for As<>() std::string Str() const ; int Int() const ; - long Long() const ; + unsigned long long U64() const ; double Double() const ; bool Bool() const ; const Array& AsArray() const ; + Array& AsArray() ; const Object& AsObject() const ; + Object& AsObject() ; // shortcuts for objects - bool Has( const std::string& key ) const ; - bool Get( const std::string& key, Val& val ) const ; - void Add( const std::string& key, const Val& val ) ; + Val& operator[]( const std::string& key ) ; // get updatable ref or throw + const Val& operator[]( const std::string& key ) const ; // get const ref or throw + Val& Item( const std::string& key ) ; // insert if not exists and get + bool Has( const std::string& key ) const ; // check if exists + bool Get( const std::string& key, Val& val ) const ; // get or return false + void Add( const std::string& key, const Val& val ) ; // insert or do nothing + void Set( const std::string& key, const Val& val ) ; // insert or update + bool Del( const std::string& key ); // delete or do nothing // shortcuts for array (and array of objects) + const Val& operator[]( std::size_t index ) const ; void Add( const Val& json ) ; std::vector Select( const std::string& key ) const ; @@ -191,7 +196,7 @@ const T& Val::As() const { try { - const Impl *impl = &dynamic_cast&>( *m_base ) ; + const Impl *impl = dynamic_cast *>( m_base.get() ) ; return impl->val ; } catch ( std::exception& e ) @@ -208,7 +213,7 @@ T& Val::As() { try { - Impl *impl = &dynamic_cast&>( *m_base ) ; + Impl *impl = dynamic_cast *>( m_base.get() ) ; return impl->val ; } catch ( std::exception& e ) diff --git a/libgrive/src/util/OS.cc b/libgrive/src/util/OS.cc index 5bf2f663..3569aa15 100644 --- a/libgrive/src/util/OS.cc +++ b/libgrive/src/util/OS.cc @@ -39,12 +39,12 @@ namespace gr { namespace os { -DateTime FileCTime( const fs::path& filename ) +void Stat( const fs::path& filename, DateTime *t, off_t *size, bool *is_dir ) { - return FileCTime( filename.string() ) ; + Stat( filename.string(), t, size, is_dir ) ; } -DateTime FileCTime( const std::string& filename ) +void Stat( const std::string& filename, DateTime *t, off_t *size, bool *is_dir ) { struct stat s = {} ; if ( ::stat( filename.c_str(), &s ) != 0 ) @@ -57,11 +57,18 @@ DateTime FileCTime( const std::string& filename ) ) ; } + if (t) + { #if defined __APPLE__ && defined __DARWIN_64_BIT_INO_T - return DateTime( s.st_ctimespec.tv_sec, s.st_ctimespec.tv_nsec ) ; + *t = DateTime( s.st_ctimespec.tv_sec, s.st_ctimespec.tv_nsec ) ; #else - return DateTime( s.st_ctim.tv_sec, s.st_ctim.tv_nsec); + *t = DateTime( s.st_ctim.tv_sec, s.st_ctim.tv_nsec); #endif + } + if (size) + *size = s.st_size; + if (is_dir) + *is_dir = S_ISDIR(s.st_mode) ? true : false; } void SetFileTime( const fs::path& filename, const DateTime& t ) diff --git a/libgrive/src/util/OS.hh b/libgrive/src/util/OS.hh index 31ab51de..edb9a12c 100644 --- a/libgrive/src/util/OS.hh +++ b/libgrive/src/util/OS.hh @@ -33,8 +33,8 @@ namespace os { struct Error : virtual Exception {} ; - DateTime FileCTime( const std::string& filename ) ; - DateTime FileCTime( const fs::path& filename ) ; + void Stat( const std::string& filename, DateTime *t, off_t *size, bool *is_dir ) ; + void Stat( const fs::path& filename, DateTime *t, off_t *size, bool *is_dir ) ; void SetFileTime( const std::string& filename, const DateTime& t ) ; void SetFileTime( const fs::path& filename, const DateTime& t ) ; diff --git a/libgrive/test/base/ResourceTest.cc b/libgrive/test/base/ResourceTest.cc index f6819034..3aebffd0 100644 --- a/libgrive/test/base/ResourceTest.cc +++ b/libgrive/test/base/ResourceTest.cc @@ -25,6 +25,7 @@ #include "drive/Entry1.hh" #include "xml/Node.hh" +#include "json/Val.hh" #include @@ -53,7 +54,8 @@ void ResourceTest::TestNormal( ) GRUT_ASSERT_EQUAL( subject.Path(), fs::path( TEST_DATA ) / "entry.xml" ) ; - subject.FromLocal( DateTime() ) ; + Val st; + subject.FromLocal( DateTime(), st ) ; GRUT_ASSERT_EQUAL( subject.MD5(), "c0742c0a32b2c909b6f176d17a6992d0" ) ; GRUT_ASSERT_EQUAL( subject.StateStr(), "local_new" ) ; From af05c7c62636cd995454134cba511e1cb373e3f2 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 1 Jan 2016 22:04:39 +0300 Subject: [PATCH 122/166] Use stored server times to check for server-side modifications (broken because deleted/new detection logic relies on last_sync/last_change) --- libgrive/src/base/Resource.cc | 112 ++++++++++++++--------------- libgrive/src/base/Resource.hh | 10 +-- libgrive/src/base/ResourceTree.cc | 4 +- libgrive/src/base/ResourceTree.hh | 2 +- libgrive/src/base/State.cc | 48 ++++--------- libgrive/src/base/State.hh | 2 +- libgrive/test/base/ResourceTest.cc | 2 +- 7 files changed, 75 insertions(+), 105 deletions(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index eb9c74c8..d7d4ba81 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -77,13 +77,13 @@ void Resource::SetState( State new_state ) boost::bind( &Resource::SetState, _1, new_state ) ) ; } -void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_change ) +void Resource::FromRemoteFolder( const Entry& remote ) { fs::path path = Path() ; if ( !remote.IsEditable() ) Log( "folder %1% is read-only", path, log::verbose ) ; - + // already sync if ( fs::is_directory( path ) ) { @@ -91,48 +91,39 @@ void Resource::FromRemoteFolder( const Entry& remote, const DateTime& last_chang m_state = sync ; } + else if ( fs::exists( path ) ) + { + // TODO: handle type change + Log( "%1% changed from folder to file", path, log::verbose ) ; + m_state = sync ; + } + // remote file created after last sync, so remote is newer - else if ( remote.MTime() > last_change ) + // TODO: Check local index instead of last_sync time + else if ( remote.MTime() > last_sync ) { - if ( fs::exists( path ) ) - { - // TODO: handle type change - Log( "%1% changed from folder to file", path, log::verbose ) ; - m_state = sync ; - } - else - { - // make all children as remote_new, if any - Log( "folder %1% is created in remote", path, log::verbose ) ; - SetState( remote_new ) ; - } + // make all children as remote_new, if any + Log( "folder %1% is created in remote", path, log::verbose ) ; + SetState( remote_new ) ; } + else { - if ( fs::exists( path ) ) - { - // TODO: handle type chage - Log( "%1% changed from file to folder", path, log::verbose ) ; - m_state = sync ; - } - else - { - Log( "folder %1% is deleted in local", path, log::verbose ) ; - SetState( local_deleted ) ; - } + Log( "folder %1% is deleted in local", path, log::verbose ) ; + SetState( local_deleted ) ; } } /// Update the state according to information (i.e. Entry) from remote. This function /// compares the modification time and checksum of both copies and determine which /// one is newer. -void Resource::FromRemote( const Entry& remote, const DateTime& last_change ) +void Resource::FromRemote( const Entry& remote ) { // sync folder if ( remote.IsDir() && IsFolder() ) - FromRemoteFolder( remote, last_change ) ; + FromRemoteFolder( remote ) ; else - FromRemoteFile( remote, last_change ) ; + FromRemoteFile( remote ) ; AssignIDs( remote ) ; @@ -156,7 +147,7 @@ void Resource::AssignIDs( const Entry& remote ) } } -void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_change ) +void Resource::FromRemoteFile( const Entry& remote ) { assert( m_parent != 0 ) ; @@ -179,7 +170,8 @@ void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_change { Trace( "file %1% change stamp = %2%", Path(), remote.ChangeStamp() ) ; - if ( remote.MTime() > last_change || remote.ChangeStamp() > 0 ) + // TODO: Check local index instead of last_sync time + if ( remote.MTime() > last_sync || remote.ChangeStamp() > 0 ) { Log( "file %1% is created in remote (change %2%)", path, remote.ChangeStamp(), log::verbose ) ; @@ -214,7 +206,7 @@ void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_change assert( m_state != unknown ) ; // if remote is modified - if ( remote.MTime() > m_mtime ) + if ( remote.MTime().Sec() > m_mtime.Sec() ) { Log( "file %1% is changed in remote", path, log::verbose ) ; m_state = remote_changed ; @@ -233,7 +225,7 @@ void Resource::FromRemoteFile( const Entry& remote, const DateTime& last_change /// Update the resource with the attributes of local file or directory. This /// function will propulate the fields in m_entry. -void Resource::FromLocal( const DateTime& last_sync, Val& state ) +void Resource::FromLocal( Val& state ) { assert( !m_json ); m_json = &state; @@ -246,37 +238,43 @@ void Resource::FromLocal( const DateTime& last_sync, Val& state ) os::Stat( path, &m_ctime, &m_size, &is_dir ) ; m_name = path.filename().string() ; - if ( !is_dir ) + m_kind = is_dir ? "folder" : "file"; + + bool is_changed; + if ( state.Has( "ctime" ) && (u64_t) m_ctime.Sec() <= state["ctime"].U64() && + ( is_dir || state.Has( "md5" ) ) ) { - m_kind = "file"; - if ( state.Has( "ctime" ) && state.Has( "md5" ) && (u64_t) m_ctime.Sec() == state["ctime"].U64() ) + if ( !is_dir ) m_md5 = state["md5"]; - else + is_changed = false; + } + else + { + if ( !is_dir ) { m_md5 = crypt::MD5::Get( path ); + // File is changed locally. TODO: Detect conflicts + is_changed = state.Has( "md5" ) && m_md5 != state["md5"].Str(); state.Set( "md5", Val( m_md5 ) ); - state.Set( "ctime", Val( m_ctime.Sec() ) ); } - if ( state.Has( "srv_time" ) && m_mtime != DateTime() ) - m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; + else + is_changed = true; + state.Set( "ctime", Val( m_ctime.Sec() ) ); } - else - { - m_kind = "folder"; + if ( state.Has( "srv_time" ) && m_mtime != DateTime() ) + m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; + if ( is_dir ) state.Del( "md5" ); - state.Del( "ctime" ); - state.Del( "srv_time" ); - } // follow parent recursively if ( m_parent->m_state == local_new || m_parent->m_state == local_deleted ) - m_state = local_new ; - - // if the file is not created after last sync, assume file is - // remote_deleted first, it will be updated to sync/remote_changed - // in FromRemote() + m_state = m_parent->m_state ; else - m_state = ( m_ctime > last_sync ? local_new : remote_deleted ) ; + { + // Upload file if it is changed and remove if not. + // State will be updated to sync/remote_changed in FromRemote() + m_state = is_changed ? local_new : remote_deleted; + } } assert( m_state != unknown ) ; @@ -389,7 +387,7 @@ Resource* Resource::FindChild( const std::string& name ) } // try to change the state to "sync" -void Resource::Sync( Syncer *syncer, DateTime& sync_time, const Val& options ) +void Resource::Sync( Syncer *syncer, const Val& options ) { assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced @@ -410,15 +408,11 @@ void Resource::Sync( Syncer *syncer, DateTime& sync_time, const Val& options ) return; } - // we want the server sync time, so we will take the server time of the last file uploaded to store as the sync time - // m_mtime is updated to server modified time when the file is uploaded - sync_time = std::max(sync_time, m_mtime); - // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) { std::for_each( m_child.begin(), m_child.end(), - boost::bind( &Resource::Sync, _1, syncer, boost::ref(sync_time), options ) ) ; + boost::bind( &Resource::Sync, _1, syncer, options ) ) ; if ( IsFolder() ) { // delete state of removed files @@ -512,7 +506,7 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) break ; } - if ( m_state != local_deleted && m_state != remote_deleted && !IsFolder() ) + if ( m_state != local_deleted && m_state != remote_deleted ) { // Update server time of this file m_json->Set( "srv_time", Val( m_mtime.Sec() ) ); diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index 99b93154..a9b5ebfd 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -106,10 +106,10 @@ public : bool HasID() const ; std::string MD5() const ; - void FromRemote( const Entry& remote, const DateTime& last_change ) ; - void FromLocal( const DateTime& last_sync, Val& state ) ; + void FromRemote( const Entry& remote ) ; + void FromLocal( Val& state ) ; - void Sync( Syncer* syncer, DateTime& sync_time, const Val& options ) ; + void Sync( Syncer* syncer, const Val& options ) ; void SetServerTime( const DateTime& time ) ; // children access @@ -129,8 +129,8 @@ private : private : void SetState( State new_state ) ; - void FromRemoteFolder( const Entry& remote, const DateTime& last_change ) ; - void FromRemoteFile( const Entry& remote, const DateTime& last_change ) ; + void FromRemoteFolder( const Entry& remote ) ; + void FromRemoteFile( const Entry& remote ) ; void DeleteLocal() ; void SetIndex() ; diff --git a/libgrive/src/base/ResourceTree.cc b/libgrive/src/base/ResourceTree.cc index 66bb379a..363c5c6f 100644 --- a/libgrive/src/base/ResourceTree.cc +++ b/libgrive/src/base/ResourceTree.cc @@ -123,11 +123,11 @@ void ResourceTree::Erase( Resource *coll ) s.erase( s.find( coll ) ) ; } -void ResourceTree::Update( Resource *coll, const Entry& e, const DateTime& last_change ) +void ResourceTree::Update( Resource *coll, const Entry& e ) { assert( coll != 0 ) ; - coll->FromRemote( e, last_change ) ; + coll->FromRemote( e ) ; ReInsert( coll ) ; } diff --git a/libgrive/src/base/ResourceTree.hh b/libgrive/src/base/ResourceTree.hh index c1945180..f5500044 100644 --- a/libgrive/src/base/ResourceTree.hh +++ b/libgrive/src/base/ResourceTree.hh @@ -75,7 +75,7 @@ public : void Insert( Resource *coll ) ; void Erase( Resource *coll ) ; - void Update( Resource *coll, const Entry& e, const DateTime& last_change ) ; + void Update( Resource *coll, const Entry& e ) ; Resource* Root() ; const Resource* Root() const ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 0cfa5f83..d677d4ba 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -65,14 +65,11 @@ State::State( const fs::path& filename, const Val& options ) : // the "-f" option will make grive always think remote is newer if ( force ) - { - m_last_change = DateTime() ; - m_last_sync = DateTime::Now() ; - } + m_last_sync = new DateTime() ; m_ign_re = boost::regex( m_ign.empty() ? "^\\.(grive|grive_state|trash)" : ( m_ign+"|^\\.(grive|grive_state|trash)" ) ); - Log( "last server change time: %1%; last sync time: %2%", m_last_change, m_last_sync, log::verbose ) ; + Log( "last sync time: %2%", m_last_sync, log::verbose ) ; } State::~State() @@ -85,7 +82,9 @@ void State::FromLocal( const fs::path& p ) { if ( !m_st.Has( "tree" ) ) m_st.Add( "tree", Val() ); - m_res.Root()->FromLocal( m_last_sync, m_st ) ; + // Remember m_update_sync just before reading local file tree + m_update_sync = DateTime::Now() ; + m_res.Root()->FromLocal( m_st ) ; FromLocal( p, m_res.Root(), m_st["tree"] ) ; } @@ -120,7 +119,7 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) if ( !tree.Has( fname ) ) tree.Add( fname, Val() ); Val& rec = tree[fname]; - c->FromLocal( m_last_sync, rec ) ; + c->FromLocal( rec ) ; if ( c->IsFolder() ) { if ( !rec.Has("tree") ) @@ -189,7 +188,7 @@ void State::FromChange( const Entry& e ) // entries in the change feed is always treated as newer in remote, // so we override the last sync time to 0 if ( Resource *res = m_res.FindByHref( e.SelfHref() ) ) - m_res.Update( res, e, DateTime() ) ; + m_res.Update( res, e ) ; } bool State::Update( const Entry& e ) @@ -205,7 +204,7 @@ bool State::Update( const Entry& e ) Log( "%1% is ignored by grive", path, log::verbose ) ; return true; } - m_res.Update( res, e, m_last_change ) ; + m_res.Update( res, e ) ; return true; } else if ( Resource *parent = m_res.FindByHref( e.ParentHref() ) ) @@ -225,7 +224,7 @@ bool State::Update( const Entry& e ) if ( child ) { // since we are updating the ID and Href, we need to remove it and re-add it. - m_res.Update( child, e, m_last_change ) ; + m_res.Update( child, e ) ; } // folder entry exist in google drive, but not local. we should create @@ -238,7 +237,7 @@ bool State::Update( const Entry& e ) m_res.Insert( child ) ; // update the state of the resource - m_res.Update( child, e, m_last_change ) ; + m_res.Update( child, e ) ; } return true ; @@ -265,16 +264,13 @@ State::iterator State::end() void State::Read( const fs::path& filename ) { m_last_sync.Assign( 0 ) ; - m_last_change.Assign( 0 ) ; try { File file( filename ) ; m_st = ParseJson( file ); Val last_sync = m_st["last_sync"] ; - Val last_change = m_st.Has( "last_change" ) ? m_st["last_change"] : m_st["last_sync"] ; m_last_sync.Assign( last_sync["sec"].Int(), last_sync["nsec"].Int() ) ; - m_last_change.Assign( last_change["sec"].Int(), last_change["nsec"].Int() ) ; m_ign = m_st.Has( "ignore_regexp" ) ? m_st["ignore_regexp"].Str() : std::string(); m_cstamp = m_st["change_stamp"].Int() ; @@ -290,12 +286,7 @@ void State::Write( const fs::path& filename ) last_sync.Add( "sec", Val( (int)m_last_sync.Sec() ) ); last_sync.Add( "nsec", Val( (unsigned)m_last_sync.NanoSec() ) ); - Val last_change ; - last_change.Add( "sec", Val( (int)m_last_change.Sec() ) ); - last_change.Add( "nsec", Val( (unsigned)m_last_change.NanoSec() ) ); - m_st.Set( "last_sync", last_sync ) ; - m_st.Set( "last_change", last_change ) ; m_st.Set( "change_stamp", Val( m_cstamp ) ) ; m_st.Set( "ignore_regexp", Val( m_ign ) ) ; @@ -305,24 +296,9 @@ void State::Write( const fs::path& filename ) void State::Sync( Syncer *syncer, const Val& options ) { - // set the last sync time from the time returned by the server for the last file synced - // if the sync time hasn't changed (i.e. now files have been uploaded) // set the last sync time to the time on the client - // ideally because we compare server file times to the last sync time - // the last sync time would always be a server time rather than a client time - // TODO - WARNING - do we use the last sync time to compare to client file times - // need to check if this introduces a new problem - DateTime last_change_time = m_last_change; - m_res.Root()->Sync( syncer, last_change_time, options ) ; - - if ( last_change_time == m_last_change ) - Trace( "nothing changed at the server side since %1%", m_last_change ) ; - else - { - Trace( "updating last server-side change time to %1%", last_change_time ) ; - m_last_change = last_change_time; - } - m_last_sync = DateTime::Now(); + m_res.Root()->Sync( syncer, options ) ; + m_last_sync = m_update_sync; } long State::ChangeStamp() const diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 353031d6..5cb85641 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -75,7 +75,7 @@ private : private : ResourceTree m_res ; DateTime m_last_sync ; - DateTime m_last_change ; + DateTime m_update_sync ; int m_cstamp ; std::string m_ign ; boost::regex m_ign_re ; diff --git a/libgrive/test/base/ResourceTest.cc b/libgrive/test/base/ResourceTest.cc index 3aebffd0..f5cdc121 100644 --- a/libgrive/test/base/ResourceTest.cc +++ b/libgrive/test/base/ResourceTest.cc @@ -63,7 +63,7 @@ void ResourceTest::TestNormal( ) entry.AddElement( "updated" ).AddText( "2012-05-09T16:13:22.401Z" ) ; Entry1 remote( entry ) ; - subject.FromRemote( remote, DateTime() ) ; + subject.FromRemote( remote ) ; GRUT_ASSERT_EQUAL( "local_changed", subject.StateStr() ) ; } From 8f640ebdad514237bcac8eb41cd9727b14f8a0f7 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 2 Jan 2016 15:04:28 +0300 Subject: [PATCH 123/166] Check local index instead of last_sync time, totally disable separate folder and changes sync passes --- libgrive/src/base/Drive.cc | 60 +++++------------------ libgrive/src/base/Drive.hh | 2 +- libgrive/src/base/Resource.cc | 76 ++++++++++++++++++------------ libgrive/src/base/Resource.hh | 7 ++- libgrive/src/base/State.cc | 32 ++++++++----- libgrive/test/base/ResourceTest.cc | 2 +- 6 files changed, 83 insertions(+), 96 deletions(-) diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index 9190111b..bd306a1b 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -58,19 +58,7 @@ Drive::Drive( Syncer *syncer, const Val& options ) : void Drive::FromRemote( const Entry& entry ) { - // entries from change feed does not have the parent HREF, - // so these checkings are done in normal entries only - Resource *parent = m_state.FindByHref( entry.ParentHref() ) ; - - if ( parent && !parent->IsFolder() ) - Log( "warning: entry %1% has parent %2% which is not a folder, ignored", - entry.Title(), parent->Name(), log::verbose ) ; - - else if ( !parent || !parent->IsInRootTree() ) - Log( "file \"%1%\" parent doesn't exist, ignored", entry.Title(), log::verbose ) ; - - else - m_state.FromRemote( entry ) ; + m_state.FromRemote( entry ) ; } void Drive::FromChange( const Entry& entry ) @@ -88,43 +76,10 @@ void Drive::SaveState() m_state.Write( m_root / state_file ) ; } -void Drive::SyncFolders( ) -{ - Log( "Synchronizing folders", log::info ) ; - - std::auto_ptr feed = m_syncer->GetFolders() ; - while ( feed->GetNext( m_syncer->Agent() ) ) - { - // first, get all collections from the query result - for ( Feed::iterator i = feed->begin() ; i != feed->end() ; ++i ) - { - const Entry &e = *i ; - if ( e.IsDir() ) - { - if ( e.ParentHrefs().size() != 1 ) - Log( "folder \"%1%\" has multiple parents, ignored", e.Title(), log::verbose ) ; - - else if ( e.Title().find('/') != std::string::npos ) - Log( "folder \"%1%\" contains a slash in its name, ignored", e.Title(), log::verbose ) ; - - else - m_state.FromRemote( e ) ; - } - } - } - - m_state.ResolveEntry() ; -} - void Drive::DetectChanges() { Log( "Reading local directories", log::info ) ; m_state.FromLocal( m_root ) ; - - long prev_stamp = m_state.ChangeStamp() ; - Trace( "previous change stamp is %1%", prev_stamp ) ; - - SyncFolders( ) ; Log( "Reading remote server file list", log::info ) ; std::auto_ptr feed = m_syncer->GetAll() ; @@ -135,12 +90,19 @@ void Drive::DetectChanges() feed->begin(), feed->end(), boost::bind( &Drive::FromRemote, this, _1 ) ) ; } - - // pull the changes feed + m_state.ResolveEntry() ; +} + +// pull the changes feed +// FIXME: unused until Grive will use the feed-based sync instead of reading full tree +void Drive::ReadChanges() +{ + long prev_stamp = m_state.ChangeStamp() ; if ( prev_stamp != -1 ) { + Trace( "previous change stamp is %1%", prev_stamp ) ; Log( "Detecting changes from last sync", log::info ) ; - feed = m_syncer->GetChanges( prev_stamp+1 ) ; + std::auto_ptr feed = m_syncer->GetChanges( prev_stamp+1 ) ; while ( feed->GetNext( m_syncer->Agent() ) ) { std::for_each( diff --git a/libgrive/src/base/Drive.hh b/libgrive/src/base/Drive.hh index db1a027c..a930192e 100644 --- a/libgrive/src/base/Drive.hh +++ b/libgrive/src/base/Drive.hh @@ -49,7 +49,7 @@ public : struct Error : virtual Exception {} ; private : - void SyncFolders( ) ; + void ReadChanges() ; void FromRemote( const Entry& entry ) ; void FromChange( const Entry& entry ) ; void UpdateChangeStamp( ) ; diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index d7d4ba81..d87c42af 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -90,23 +90,18 @@ void Resource::FromRemoteFolder( const Entry& remote ) Log( "folder %1% is in sync", path, log::verbose ) ; m_state = sync ; } - else if ( fs::exists( path ) ) { // TODO: handle type change Log( "%1% changed from folder to file", path, log::verbose ) ; m_state = sync ; } - - // remote file created after last sync, so remote is newer - // TODO: Check local index instead of last_sync time - else if ( remote.MTime() > last_sync ) + else if ( remote.MTime().Sec() > m_mtime.Sec() ) // FIXME only seconds are stored in local index { - // make all children as remote_new, if any + // remote folder created after last sync, so remote is newer Log( "folder %1% is created in remote", path, log::verbose ) ; SetState( remote_new ) ; } - else { Log( "folder %1% is deleted in local", path, log::verbose ) ; @@ -131,6 +126,7 @@ void Resource::FromRemote( const Entry& remote ) if ( m_state == remote_new || m_state == remote_changed ) m_md5 = remote.MD5() ; + m_mtime = remote.MTime() ; } @@ -170,8 +166,7 @@ void Resource::FromRemoteFile( const Entry& remote ) { Trace( "file %1% change stamp = %2%", Path(), remote.ChangeStamp() ) ; - // TODO: Check local index instead of last_sync time - if ( remote.MTime() > last_sync || remote.ChangeStamp() > 0 ) + if ( remote.MTime().Sec() > m_mtime.Sec() || remote.MD5() != m_md5 || remote.ChangeStamp() > 0 ) { Log( "file %1% is created in remote (change %2%)", path, remote.ChangeStamp(), log::verbose ) ; @@ -223,6 +218,19 @@ void Resource::FromRemoteFile( const Entry& remote ) } } +void Resource::FromDeleted( Val& state ) +{ + assert( !m_json ); + m_json = &state; + if ( state.Has( "ctime" ) ) + m_ctime.Assign( state["ctime"].U64(), 0 ); + if ( state.Has( "md5" ) ) + m_md5 = state["md5"]; + if ( state.Has( "srv_time" ) ) + m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; + m_state = both_deleted; +} + /// Update the resource with the attributes of local file or directory. This /// function will propulate the fields in m_entry. void Resource::FromLocal( Val& state ) @@ -235,7 +243,7 @@ void Resource::FromLocal( Val& state ) { fs::path path = Path() ; bool is_dir; - os::Stat( path, &m_ctime, &m_size, &is_dir ) ; + os::Stat( path, &m_ctime, NULL, &is_dir ) ; m_name = path.filename().string() ; m_kind = is_dir ? "folder" : "file"; @@ -254,7 +262,7 @@ void Resource::FromLocal( Val& state ) { m_md5 = crypt::MD5::Get( path ); // File is changed locally. TODO: Detect conflicts - is_changed = state.Has( "md5" ) && m_md5 != state["md5"].Str(); + is_changed = !state.Has( "md5" ) || m_md5 != state["md5"].Str(); state.Set( "md5", Val( m_md5 ) ); } else @@ -265,9 +273,11 @@ void Resource::FromLocal( Val& state ) m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; if ( is_dir ) state.Del( "md5" ); + else + state.Del( "tree" ); // follow parent recursively - if ( m_parent->m_state == local_new || m_parent->m_state == local_deleted ) + if ( m_parent->m_state == local_new || m_parent->m_state == remote_deleted ) m_state = m_parent->m_state ; else { @@ -413,22 +423,6 @@ void Resource::Sync( Syncer *syncer, const Val& options ) { std::for_each( m_child.begin(), m_child.end(), boost::bind( &Resource::Sync, _1, syncer, options ) ) ; - if ( IsFolder() ) - { - // delete state of removed files - Val& tree = (*m_json)["tree"]; - Val::Object leftover = tree.AsObject(); - for( iterator i = m_child.begin(); i != m_child.end(); i++ ) - { - Resource *r = *i; - if ( r->m_state != local_deleted && r->m_state != remote_deleted ) - leftover.erase( r->Name() ); - else - r->m_json = NULL; - } - for( Val::Object::iterator i = leftover.begin(); i != leftover.end(); i++ ) - tree.Del( i->first ); - } } } @@ -454,7 +448,10 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) case local_deleted : Log( "sync %1% deleted in local. deleting remote", path, log::info ) ; if ( syncer ) + { syncer->DeleteRemote( this ) ; + DeleteIndex() ; + } break ; case local_changed : @@ -490,7 +487,15 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) case remote_deleted : Log( "sync %1% deleted in remote. deleting local", path, log::info ) ; if ( syncer ) + { DeleteLocal() ; + DeleteIndex() ; + } + break ; + + case both_deleted : + if ( syncer ) + DeleteIndex() ; break ; case sync : @@ -506,7 +511,7 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) break ; } - if ( m_state != local_deleted && m_state != remote_deleted ) + if ( syncer && m_json ) { // Update server time of this file m_json->Set( "srv_time", Val( m_mtime.Sec() ) ); @@ -548,18 +553,27 @@ void Resource::DeleteLocal() } } +void Resource::DeleteIndex() +{ + (*m_parent->m_json)["tree"].Del( Name() ); + m_json = NULL; +} + void Resource::SetIndex() { assert( m_parent->m_json != NULL ); if ( !m_json ) m_json = &((*m_parent->m_json)["tree"]).Item( Name() ); bool is_dir; - os::Stat( Path(), &m_ctime, &m_size, &is_dir ); + os::Stat( Path(), &m_ctime, NULL, &is_dir ); if ( !is_dir ) { m_json->Set( "ctime", Val( m_ctime.Sec() ) ); m_json->Set( "md5", Val( m_md5 ) ); + m_json->Del( "tree" ); } + else // check if tree item exists + m_json->Item( "tree" ); } Resource::iterator Resource::begin() const @@ -582,7 +596,7 @@ std::ostream& operator<<( std::ostream& os, Resource::State s ) static const char *state[] = { "sync", "local_new", "local_changed", "local_deleted", "remote_new", - "remote_changed", "remote_deleted" + "remote_changed", "remote_deleted", "both_deleted" } ; assert( s >= 0 && s < Count(state) ) ; return os << state[s] ; diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index a9b5ebfd..97c55d4e 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -67,12 +67,14 @@ public : /// We should download the file. remote_new, - /// Resource exists in both local & remote, but remote is newer. + /// Resource exists in both local & remote, but remote is newer. remote_changed, /// Resource delete in remote, need to delete in local remote_deleted, + /// Both deleted. State is used to remove leftover files from the index after sync. + both_deleted, /// invalid value unknown @@ -107,6 +109,7 @@ public : std::string MD5() const ; void FromRemote( const Entry& remote ) ; + void FromDeleted( Val& state ) ; void FromLocal( Val& state ) ; void Sync( Syncer* syncer, const Val& options ) ; @@ -133,6 +136,7 @@ private : void FromRemoteFile( const Entry& remote ) ; void DeleteLocal() ; + void DeleteIndex() ; void SetIndex() ; void SyncSelf( Syncer* syncer, const Val& options ) ; @@ -143,7 +147,6 @@ private : std::string m_md5 ; DateTime m_mtime ; DateTime m_ctime ; - off_t m_size ; std::string m_id ; std::string m_href ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index d677d4ba..9d122329 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -65,7 +65,7 @@ State::State( const fs::path& filename, const Val& options ) : // the "-f" option will make grive always think remote is newer if ( force ) - m_last_sync = new DateTime() ; + m_last_sync = DateTime() ; m_ign_re = boost::regex( m_ign.empty() ? "^\\.(grive|grive_state|trash)" : ( m_ign+"|^\\.(grive|grive_state|trash)" ) ); @@ -80,12 +80,10 @@ State::~State() /// of local directory. void State::FromLocal( const fs::path& p ) { - if ( !m_st.Has( "tree" ) ) - m_st.Add( "tree", Val() ); // Remember m_update_sync just before reading local file tree m_update_sync = DateTime::Now() ; m_res.Root()->FromLocal( m_st ) ; - FromLocal( p, m_res.Root(), m_st["tree"] ) ; + FromLocal( p, m_res.Root(), m_st.Item( "tree" ) ) ; } bool State::IsIgnore( const std::string& filename ) @@ -98,6 +96,8 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) assert( folder != 0 ) ; assert( folder->IsFolder() ) ; + Val::Object leftover = tree.AsObject(); + for ( fs::directory_iterator i( p ) ; i != fs::directory_iterator() ; ++i ) { std::string fname = i->path().filename().string() ; @@ -116,17 +116,25 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) folder->AddChild( c ) ; m_res.Insert( c ) ; } - if ( !tree.Has( fname ) ) - tree.Add( fname, Val() ); - Val& rec = tree[fname]; + leftover.erase( fname ); + Val& rec = tree.Item( fname ); c->FromLocal( rec ) ; if ( c->IsFolder() ) - { - if ( !rec.Has("tree") ) - rec.Add( "tree", Val() ); - FromLocal( *i, c, rec["tree"] ) ; - } + FromLocal( *i, c, rec.Item( "tree" ) ) ; + } + } + + for( Val::Object::iterator i = leftover.begin(); i != leftover.end(); i++ ) + { + // Restore state of locally deleted files + Resource *c = folder->FindChild( i->first ) ; + if ( !c ) + { + c = new Resource( i->first, i->second.Has( "tree" ) ? "folder" : "file" ) ; + folder->AddChild( c ) ; + m_res.Insert( c ) ; } + c->FromDeleted( tree.Item( i->first ) ); } } diff --git a/libgrive/test/base/ResourceTest.cc b/libgrive/test/base/ResourceTest.cc index f5cdc121..cbbe6c86 100644 --- a/libgrive/test/base/ResourceTest.cc +++ b/libgrive/test/base/ResourceTest.cc @@ -55,7 +55,7 @@ void ResourceTest::TestNormal( ) GRUT_ASSERT_EQUAL( subject.Path(), fs::path( TEST_DATA ) / "entry.xml" ) ; Val st; - subject.FromLocal( DateTime(), st ) ; + subject.FromLocal( st ) ; GRUT_ASSERT_EQUAL( subject.MD5(), "c0742c0a32b2c909b6f176d17a6992d0" ) ; GRUT_ASSERT_EQUAL( subject.StateStr(), "local_new" ) ; From 23fa985bdb6ea0265ddcfe96eea87d6d007aa232 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 2 Jan 2016 15:05:37 +0300 Subject: [PATCH 124/166] Do not track last_sync at all, clear srv_time's with -f option --- CMakeLists.txt | 2 +- README.md | 9 +++++++-- libgrive/src/base/State.cc | 31 ++++++++++--------------------- libgrive/src/base/State.hh | 3 +-- 4 files changed, 19 insertions(+), 26 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index d0ba0d54..fac9c1af 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! -set( GRIVE_VERSION "0.4.2" ) +set( GRIVE_VERSION "0.5-pre" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) diff --git a/README.md b/README.md index ca6e53e8..8d4bea47 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# Grive2 0.4.2 +# Grive2 0.5-dev -28 Dec 2015, Vitaliy Filippov +02 Jan 2016, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 @@ -70,6 +70,11 @@ Enjoy! ## Version History +### Grive2 v0.5 (unreleased) + +- Much faster and more correct synchronisation using local modification time and checksum cache (similar to git index) +- force option works again + ### Grive2 v0.4.2 - Option to exclude files by perl regexp diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 9d122329..8f61b599 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -38,14 +38,15 @@ State::State( const fs::path& filename, const Val& options ) : { Read( filename ) ; - bool force = options.Has( "force" ) ? options["force"].Bool() : false ; + // the "-f" option will make grive always think remote is newer + m_force = options.Has( "force" ) ? options["force"].Bool() : false ; if ( options.Has( "ignore" ) && options["ignore"].Str() != m_ign ) { // also "-f" is implicitly turned on when ignore regexp is changed // because without it grive would think that previously ignored files are deleted locally if ( !m_ign.empty() ) - force = true; + m_force = true; m_ign = options["ignore"].Str(); } else if ( options.Has( "dir" ) ) @@ -58,18 +59,12 @@ State::State( const fs::path& filename, const Val& options ) : const boost::regex esc( "[.^$|()\\[\\]{}*+?\\\\]" ); std::string ign = "^(?!"+regex_replace( m_dir, esc, "\\\\&", boost::format_sed )+"(/|$))"; if ( !m_ign.empty() && ign != m_ign ) - force = true; + m_force = true; m_ign = ign; } } - // the "-f" option will make grive always think remote is newer - if ( force ) - m_last_sync = DateTime() ; - m_ign_re = boost::regex( m_ign.empty() ? "^\\.(grive|grive_state|trash)" : ( m_ign+"|^\\.(grive|grive_state|trash)" ) ); - - Log( "last sync time: %2%", m_last_sync, log::verbose ) ; } State::~State() @@ -80,8 +75,6 @@ State::~State() /// of local directory. void State::FromLocal( const fs::path& p ) { - // Remember m_update_sync just before reading local file tree - m_update_sync = DateTime::Now() ; m_res.Root()->FromLocal( m_st ) ; FromLocal( p, m_res.Root(), m_st.Item( "tree" ) ) ; } @@ -118,6 +111,8 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) } leftover.erase( fname ); Val& rec = tree.Item( fname ); + if ( m_force ) + rec.Del( "srv_time" ); c->FromLocal( rec ) ; if ( c->IsFolder() ) FromLocal( *i, c, rec.Item( "tree" ) ) ; @@ -134,7 +129,10 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) folder->AddChild( c ) ; m_res.Insert( c ) ; } - c->FromDeleted( tree.Item( i->first ) ); + Val& rec = tree.Item( i->first ); + if ( m_force ) + rec.Del( "srv_time" ); + c->FromDeleted( rec ); } } @@ -271,14 +269,11 @@ State::iterator State::end() void State::Read( const fs::path& filename ) { - m_last_sync.Assign( 0 ) ; try { File file( filename ) ; m_st = ParseJson( file ); - Val last_sync = m_st["last_sync"] ; - m_last_sync.Assign( last_sync["sec"].Int(), last_sync["nsec"].Int() ) ; m_ign = m_st.Has( "ignore_regexp" ) ? m_st["ignore_regexp"].Str() : std::string(); m_cstamp = m_st["change_stamp"].Int() ; @@ -290,11 +285,6 @@ void State::Read( const fs::path& filename ) void State::Write( const fs::path& filename ) { - Val last_sync ; - last_sync.Add( "sec", Val( (int)m_last_sync.Sec() ) ); - last_sync.Add( "nsec", Val( (unsigned)m_last_sync.NanoSec() ) ); - - m_st.Set( "last_sync", last_sync ) ; m_st.Set( "change_stamp", Val( m_cstamp ) ) ; m_st.Set( "ignore_regexp", Val( m_ign ) ) ; @@ -306,7 +296,6 @@ void State::Sync( Syncer *syncer, const Val& options ) { // set the last sync time to the time on the client m_res.Root()->Sync( syncer, options ) ; - m_last_sync = m_update_sync; } long State::ChangeStamp() const diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 5cb85641..939a3716 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -74,12 +74,11 @@ private : private : ResourceTree m_res ; - DateTime m_last_sync ; - DateTime m_update_sync ; int m_cstamp ; std::string m_ign ; boost::regex m_ign_re ; Val m_st ; + bool m_force ; std::vector m_unresolved ; } ; From 00311e83654fe74a87bc7ad8b03e7486909dd792 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 3 Jan 2016 01:57:54 +0300 Subject: [PATCH 125/166] Implement automatic rename detection --- README.md | 3 +- grive/src/main.cc | 16 -------- libgrive/src/base/Drive.cc | 5 --- libgrive/src/base/Drive.hh | 1 - libgrive/src/base/Resource.cc | 55 ++++++++++++++++++++++++--- libgrive/src/base/Resource.hh | 6 ++- libgrive/src/base/ResourceTree.cc | 8 ++++ libgrive/src/base/ResourceTree.hh | 10 ++--- libgrive/src/base/State.cc | 63 +------------------------------ libgrive/src/base/State.hh | 1 - libgrive/src/drive2/Syncer2.cc | 8 +++- 11 files changed, 76 insertions(+), 100 deletions(-) diff --git a/README.md b/README.md index 8d4bea47..756873b8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Grive2 0.5-dev -02 Jan 2016, Vitaliy Filippov +3 Jan 2016, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 @@ -73,6 +73,7 @@ Enjoy! ### Grive2 v0.5 (unreleased) - Much faster and more correct synchronisation using local modification time and checksum cache (similar to git index) +- Automatic move/rename detection, -m option removed - force option works again ### Grive2 v0.4.2 diff --git a/grive/src/main.cc b/grive/src/main.cc index 347f3235..b0fb4ae8 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -121,7 +121,6 @@ int Main( int argc, char **argv ) ( "dry-run", "Only detect which files need to be uploaded/downloaded, " "without actually performing them." ) ( "ignore", po::value(), "Perl RegExp to ignore files (matched against relative paths, remembered for next runs)." ) - ( "move,m", po::value >()->multitoken(), "Syncs, then moves a file (first argument) to new location (second argument) without reuploading or redownloading." ) ; po::variables_map vm; @@ -205,21 +204,6 @@ int Main( int argc, char **argv ) else drive.DryRun() ; - if ( vm.count ( "move" ) > 0 && vm.count( "dry-run" ) == 0 ) - { - if (vm["move"].as >().size() < 2 ) - Log( "Not enough arguments for move. Move failed.", log::error ); - else - { - bool success = drive.Move( vm["move"].as >()[0], - vm["move"].as >()[1] ); - if (success) - Log( "Move successful!", log::info ); - else - Log( "Move failed.", log::error); - } - } - config.Save() ; Log( "Finished!", log::info ) ; return 0 ; diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index bd306a1b..9dd80899 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -112,11 +112,6 @@ void Drive::ReadChanges() } } -bool Drive::Move( fs::path old_p, fs::path new_p ) -{ - return m_state.Move( m_syncer, old_p, new_p, m_options["path"].Str() ); -} - void Drive::Update() { Log( "Synchronizing files", log::info ) ; diff --git a/libgrive/src/base/Drive.hh b/libgrive/src/base/Drive.hh index a930192e..5f99be24 100644 --- a/libgrive/src/base/Drive.hh +++ b/libgrive/src/base/Drive.hh @@ -41,7 +41,6 @@ public : Drive( Syncer *syncer, const Val& options ) ; void DetectChanges() ; - bool Move( fs::path old_p, fs::path new_p ); void Update() ; void DryRun() ; void SaveState() ; diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index d87c42af..66a4ac32 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -18,6 +18,7 @@ */ #include "Resource.hh" +#include "ResourceTree.hh" #include "Entry.hh" #include "Syncer.hh" @@ -269,7 +270,7 @@ void Resource::FromLocal( Val& state ) is_changed = true; state.Set( "ctime", Val( m_ctime.Sec() ) ); } - if ( state.Has( "srv_time" ) && m_mtime != DateTime() ) + if ( state.Has( "srv_time" ) ) m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; if ( is_dir ) state.Del( "md5" ); @@ -397,14 +398,14 @@ Resource* Resource::FindChild( const std::string& name ) } // try to change the state to "sync" -void Resource::Sync( Syncer *syncer, const Val& options ) +void Resource::Sync( Syncer *syncer, ResourceTree *res_tree, const Val& options ) { assert( m_state != unknown ) ; assert( !IsRoot() || m_state == sync ) ; // root folder is already synced try { - SyncSelf( syncer, options ) ; + SyncSelf( syncer, res_tree, options ) ; } catch ( File::Error &e ) { @@ -422,11 +423,11 @@ void Resource::Sync( Syncer *syncer, const Val& options ) if ( m_state != local_deleted && m_state != remote_deleted ) { std::for_each( m_child.begin(), m_child.end(), - boost::bind( &Resource::Sync, _1, syncer, options ) ) ; + boost::bind( &Resource::Sync, _1, syncer, res_tree, options ) ) ; } } -void Resource::SyncSelf( Syncer* syncer, const Val& options ) +void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& options ) { assert( !IsRoot() || m_state == sync ) ; // root is always sync assert( IsRoot() || !syncer || m_parent->IsFolder() ) ; @@ -435,6 +436,50 @@ void Resource::SyncSelf( Syncer* syncer, const Val& options ) const fs::path path = Path() ; + // Detect renames + if ( !IsFolder() && ( m_state == local_new || m_state == local_deleted || + m_state == remote_new || m_state == remote_deleted ) ) + { + details::MD5Range moved = res_tree->FindByMD5( m_md5 ); + bool is_local = m_state == local_new || m_state == local_deleted; + State other; + if ( m_state == local_new ) + other = local_deleted; + else if ( m_state == local_deleted ) + other = local_new; + else if ( m_state == remote_new ) + other = remote_deleted; + else + other = remote_new; + for ( details::MD5Map::iterator i = moved.first ; i != moved.second; i++ ) + { + Resource *m = *i; + if ( m->m_state == other ) + { + Resource* from = m_state == local_new || m_state == remote_new ? m : this; + Resource* to = m_state == local_new || m_state == remote_new ? this : m; + Log( "sync %1% moved to %2%. moving %3%", from->Path(), to->Path(), + is_local ? "remote" : "local", log::info ); + if ( syncer ) + { + if ( is_local ) + syncer->Move( from, to->Parent(), to->Name() ); + else + { + fs::rename( from->Path(), to->Path() ); + to->SetIndex(); + } + to->m_mtime = from->m_mtime; + to->m_json->Set( "srv_time", Val( from->m_mtime.Sec() ) ); + from->DeleteIndex(); + } + from->m_state = both_deleted; + to->m_state = sync; + return; + } + } + } + switch ( m_state ) { case local_new : diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index 97c55d4e..354369fd 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -29,6 +29,8 @@ namespace gr { +class ResourceTree ; + class Syncer ; class Val ; @@ -112,7 +114,7 @@ public : void FromDeleted( Val& state ) ; void FromLocal( Val& state ) ; - void Sync( Syncer* syncer, const Val& options ) ; + void Sync( Syncer* syncer, ResourceTree *res_tree, const Val& options ) ; void SetServerTime( const DateTime& time ) ; // children access @@ -139,7 +141,7 @@ private : void DeleteIndex() ; void SetIndex() ; - void SyncSelf( Syncer* syncer, const Val& options ) ; + void SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& options ) ; private : std::string m_name ; diff --git a/libgrive/src/base/ResourceTree.cc b/libgrive/src/base/ResourceTree.cc index 363c5c6f..16ac5347 100644 --- a/libgrive/src/base/ResourceTree.cc +++ b/libgrive/src/base/ResourceTree.cc @@ -97,6 +97,14 @@ const Resource* ResourceTree::FindByHref( const std::string& href ) const return i != map.end() ? *i : 0 ; } +MD5Range ResourceTree::FindByMD5( const std::string& md5 ) +{ + MD5Map& map = m_set.get() ; + if ( !md5.empty() ) + return map.equal_range( md5 ); + return MD5Range( map.end(), map.end() ) ; +} + /// Reinsert should be called when the ID/HREF were updated bool ResourceTree::ReInsert( Resource *coll ) { diff --git a/libgrive/src/base/ResourceTree.hh b/libgrive/src/base/ResourceTree.hh index f5500044..b1fc7285 100644 --- a/libgrive/src/base/ResourceTree.hh +++ b/libgrive/src/base/ResourceTree.hh @@ -33,7 +33,7 @@ namespace gr { namespace details { using namespace boost::multi_index ; - struct ByID {} ; + struct ByMD5 {} ; struct ByHref {} ; struct ByIdentity {} ; @@ -41,14 +41,15 @@ namespace details Resource*, indexed_by< hashed_non_unique, const_mem_fun >, - hashed_non_unique, const_mem_fun >, + hashed_non_unique, const_mem_fun >, hashed_unique, identity > > > Folders ; - typedef Folders::index::type IDMap ; + typedef Folders::index::type MD5Map ; typedef Folders::index::type HrefMap ; typedef Folders::index::type Set ; + typedef std::pair MD5Range ; } /*! \brief A simple container for storing folders @@ -68,9 +69,8 @@ public : Resource* FindByHref( const std::string& href ) ; const Resource* FindByHref( const std::string& href ) const ; + details::MD5Range FindByMD5( const std::string& md5 ) ; - Resource* FindByID( const std::string& id ) ; - bool ReInsert( Resource *coll ) ; void Insert( Resource *coll ) ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 8f61b599..f743033e 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -295,7 +295,7 @@ void State::Write( const fs::path& filename ) void State::Sync( Syncer *syncer, const Val& options ) { // set the last sync time to the time on the client - m_res.Root()->Sync( syncer, options ) ; + m_res.Root()->Sync( syncer, &m_res, options ) ; } long State::ChangeStamp() const @@ -309,65 +309,4 @@ void State::ChangeStamp( long cstamp ) m_cstamp = cstamp ; } -bool State::Move( Syncer* syncer, fs::path old_p, fs::path new_p, fs::path grive_root ) -{ - // Convert paths to canonical representations - // Also seems to remove trailing / at the end of directory paths - old_p = fs::canonical( old_p ); - grive_root = fs::canonical( grive_root ); - - // new_p is a little special because fs::canonical() requires that the path exists - if ( new_p.string()[ new_p.string().size() - 1 ] == '/') // If new_p ends with a /, remove it - new_p = new_p.parent_path(); - new_p = fs::canonical( new_p.parent_path() ) / new_p.filename(); - - // Fails if source file doesn't exist, or if destination file already - // exists and is not a directory, or if the source and destination are exactly the same - if ( (fs::exists(new_p) && !fs::is_directory(new_p)) || !fs::exists(old_p) || fs::equivalent( old_p, new_p ) ) - return false; - - // If new path is an existing directory, move the file into the directory - // instead of trying to rename it - if ( fs::is_directory( new_p ) ) - new_p = new_p / old_p.filename(); - - // Get the paths relative to grive root. - // Just finds the substring from the end of the grive_root to the end of the path - // +1s are to exclude slash at beginning of relative path - std::string root( grive_root.string() + "/" ); - if ( new_p.string().substr( 0, root.length() ).compare( root ) != 0 || - old_p.string().substr( 0, root.length() ).compare( root ) != 0 ) - return false; - fs::path new_p_rootrel( new_p.string().substr( root.length() ) ); - fs::path old_p_rootrel( old_p.string().substr( root.length() ) ); - - //Get resources - Resource* res = m_res.Root(); - Resource* newParentRes = m_res.Root(); - for ( fs::path::iterator it = old_p_rootrel.begin(); it != old_p_rootrel.end(); ++it ) - { - if ( *it != "." && *it != ".." && res ) - res = res->FindChild(it->string()); - if ( *it == ".." ) - res = res->Parent(); - } - for ( fs::path::iterator it = new_p_rootrel.begin(); it != new_p_rootrel.end(); ++it ) - { - if ( *it != "." && *it != ".." && *it != new_p.filename() && newParentRes ) - newParentRes = newParentRes->FindChild(it->string()); - if ( *it == "..") - res = res->Parent(); - } - - //These conditions should only occur if everything is not up-to-date - if ( res == 0 || newParentRes == 0 || res->GetState() != Resource::sync || - newParentRes->GetState() != Resource::sync || - newParentRes->FindChild( new_p.filename().string() ) != 0 ) - return false; - - fs::rename(old_p, new_p); //Moves local file - syncer->Move(res, newParentRes, new_p.filename().string()); //Moves server file - return true; -} - } // end of namespace gr diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 939a3716..1d5dcc1f 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -62,7 +62,6 @@ public : long ChangeStamp() const ; void ChangeStamp( long cstamp ) ; - bool Move( Syncer* syncer, fs::path old_p, fs::path new_p, fs::path grive_root ); private : void FromLocal( const fs::path& p, Resource *folder, Val& tree ) ; diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index 9d77fa50..66ad64ff 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -121,11 +121,15 @@ bool Syncer2::Move( Resource* res, Resource* newParentRes, std::string newFilena http::Header hdr2 ; hdr2.Add( "Content-Type: application/json" ); http::ValResponse vrsp ; - //Don't change modified date because we're only moving - long http_code = m_http->Put( feeds::files + "/" + res->ResourceID() + "?modifiedDateBehavior=noChange" + addRemoveParents, json_meta, &vrsp, hdr2 ) ; + // Don't change modified date because we're only moving + long http_code = m_http->Put( + feeds::files + "/" + res->ResourceID() + "?modifiedDateBehavior=noChange" + addRemoveParents, + json_meta, &vrsp, hdr2 + ) ; valr = vrsp.Response(); assert( http_code == 200 && !( valr["id"].Str().empty() ) ); } + return true; } From c647c5f89f60e784ff326b8fa325c8bb78c6000a Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 3 Jan 2016 12:51:38 +0300 Subject: [PATCH 126/166] Update debian package version --- debian/changelog | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/debian/changelog b/debian/changelog index 7ddc1e34..39d29bda 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +grive2 (0.5+git20160103) unstable; urgency=medium + + * Newer release, with support for faster sync and rename detection + + -- Vitaliy Filippov Sun, 03 Jan 2016 12:51:55 +0300 + grive2 (0.4.1+git20151011) unstable; urgency=medium * Add Debian packaging scripts to the official repository From b49a89ad343dad6a913b72750ebd16f9c548d440 Mon Sep 17 00:00:00 2001 From: Josua Mayer Date: Sun, 3 Jan 2016 14:24:55 +0100 Subject: [PATCH 127/166] std::vector::erase is expensive; use std::list:erase instead erasing from a vector at random is very expensive. std::list however is made for that purpose! --- libgrive/src/base/State.cc | 6 +++--- libgrive/src/base/State.hh | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index f743033e..7b819932 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -172,9 +172,9 @@ std::size_t State::TryResolveEntry() assert( !m_unresolved.empty() ) ; std::size_t count = 0 ; - std::vector& en = m_unresolved ; - - for ( std::vector::iterator i = en.begin() ; i != en.end() ; ) + std::list& en = m_unresolved ; + + for ( std::list::iterator i = en.begin() ; i != en.end() ; ) { if ( Update( *i ) ) { diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 1d5dcc1f..088be382 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -79,7 +79,7 @@ private : Val m_st ; bool m_force ; - std::vector m_unresolved ; + std::list m_unresolved ; } ; } // end of namespace gr From ca4a0b6b80d1e52529f25076605de430af3abea9 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 3 Jan 2016 17:40:23 +0300 Subject: [PATCH 128/166] Index allows us to not turn -f on when changing ignore regexp --- libgrive/src/base/State.cc | 8 -------- 1 file changed, 8 deletions(-) diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index f743033e..20b13ef8 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -42,13 +42,7 @@ State::State( const fs::path& filename, const Val& options ) : m_force = options.Has( "force" ) ? options["force"].Bool() : false ; if ( options.Has( "ignore" ) && options["ignore"].Str() != m_ign ) - { - // also "-f" is implicitly turned on when ignore regexp is changed - // because without it grive would think that previously ignored files are deleted locally - if ( !m_ign.empty() ) - m_force = true; m_ign = options["ignore"].Str(); - } else if ( options.Has( "dir" ) ) { const boost::regex trim_path( "^/+|/+$" ); @@ -58,8 +52,6 @@ State::State( const fs::path& filename, const Val& options ) : // "-s" is internally converted to an ignore regexp const boost::regex esc( "[.^$|()\\[\\]{}*+?\\\\]" ); std::string ign = "^(?!"+regex_replace( m_dir, esc, "\\\\&", boost::format_sed )+"(/|$))"; - if ( !m_ign.empty() && ign != m_ign ) - m_force = true; m_ign = ign; } } From 3b9aa4f2aae3d2ad2660fbd334e17b3246466ca4 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 3 Jan 2016 18:04:08 +0300 Subject: [PATCH 129/166] Add new_rev switch support to drive v2 api client --- libgrive/src/drive2/Syncer2.cc | 9 +++++---- libgrive/src/drive2/Syncer2.hh | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index 66ad64ff..ae6bc4d4 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -70,7 +70,7 @@ bool Syncer2::EditContent( Resource *res, bool new_rev ) return false ; } - return Upload( res ) ; + return Upload( res, new_rev ) ; } bool Syncer2::Create( Resource *res ) @@ -86,7 +86,7 @@ bool Syncer2::Create( Resource *res ) return false ; } - return Upload( res ); + return Upload( res, false ); } bool Syncer2::Move( Resource* res, Resource* newParentRes, std::string newFilename ) @@ -140,7 +140,7 @@ std::string to_string( uint64_t n ) return s.str(); } -bool Syncer2::Upload( Resource *res ) +bool Syncer2::Upload( Resource *res, bool new_rev ) { Val meta; meta.Add( "title", Val( res->Name() ) ); @@ -196,7 +196,8 @@ bool Syncer2::Upload( Resource *res ) http::ValResponse vrsp; m_http->Request( res->ResourceID().empty() ? "POST" : "PUT", - upload_base + ( res->ResourceID().empty() ? "" : "/" + res->ResourceID() ) + "?uploadType=multipart", + upload_base + ( res->ResourceID().empty() ? "" : "/" + res->ResourceID() ) + + "?uploadType=multipart&newRevision=" + ( new_rev ? "true" : "false" ), &multipart, &vrsp, hdr ) ; valr = vrsp.Response() ; diff --git a/libgrive/src/drive2/Syncer2.hh b/libgrive/src/drive2/Syncer2.hh index 96f93fce..8a34757e 100644 --- a/libgrive/src/drive2/Syncer2.hh +++ b/libgrive/src/drive2/Syncer2.hh @@ -46,7 +46,7 @@ public : private : - bool Upload( Resource *res ); + bool Upload( Resource *res, bool new_rev ); } ; From 9d8c77d0bd88f5fafef70c703cf55303cfa9b2ed Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 3 Jan 2016 18:16:09 +0300 Subject: [PATCH 130/166] Update manpage --- grive/doc/grive.1 | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/grive/doc/grive.1 b/grive/doc/grive.1 index 57b2e9cc..96efbff3 100644 --- a/grive/doc/grive.1 +++ b/grive/doc/grive.1 @@ -2,7 +2,7 @@ .\" First parameter, NAME, should be all caps .\" Second parameter, SECTION, should be 1-8, maybe w/ subsection .\" other parameters are allowed: see man(7), man(1) -.TH "GRIVE" 1 "June 19, 2012" +.TH "GRIVE" 1 "January 3, 2016" .SH NAME grive \- Google Drive client for GNU/Linux @@ -26,7 +26,7 @@ Requests authorization token from Google Enable debug level messages. Implies \-V .TP \fB\-\-dry-run\fR -Only detects which files are needed for download or upload without doing it +Only detect which files need to be uploaded/downloaded, without actually performing changes .TP \fB\-f, \-\-force\fR Forces @@ -36,9 +36,30 @@ to always download a file from Google Drive instead uploading it \fB\-h\fR, \fB\-\-help\fR Produces help message .TP -\fB\-l\fR filename, \fB\-\-log\fR filename -Set log output to -.I filename +\fB\-\-ignore\fR +Ignore files with relative paths matching this Perl Regular Expression. +Value is remembered for next runs. +.TP +\fB\-l\fR , \fB\-\-log\fR +Write log output to +.I +.TP +\fB\-\-log\-http\fR +Log all HTTP responses in files named +.I YYYY-MM-DD.HHMMSS.txt +for debugging +.TP +\fB\-\-new\-rev\fR +Create new revisions in server for updated files +.TP +\fB\-p\fR , \fB\-\-path\fR +Set root sync directory to +.I +.TP +\fB\-s\fR , \fB\-\-dir\fR +Sync a single +.I +subdirectory. Internally converted to an ignore regexp, remembered for next runs. .TP \fB\-v\fR, \fB\-\-version\fR Displays program version @@ -46,13 +67,17 @@ Displays program version \fB\-V\fR, \fB\-\-verbose\fR Verbose mode. Enables more messages than usual. -.SH AUTHOR +.SH AUTHORS .PP -The software was developed by Nestal Wan. +Current maintainer is Vitaliy Filippov. .PP +Original author was Nestal Wan. This manpage was written by José Luis Segura Lucas (josel.segura@gmx.es) +.PP +The full list of contributors may be found here +.I http://yourcmc.ru/wiki/Grive2#Full_list_of_contributors .SH REPORT BUGS .PP -.I https://github.com/Grive/grive +.I https://github.com/vitalif/grive2/issues .I https://groups.google.com/forum/?fromgroups#!forum/grive-devel From 0112330c1d1ca6b7bbeb59e3a48632c68c466ee3 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Tue, 5 Jan 2016 20:09:53 +0300 Subject: [PATCH 131/166] Do not die and do not update local state of file on sync exceptions. Should fix #47 --- README.md | 1 + libgrive/src/base/Drive.cc | 1 - libgrive/src/base/Resource.cc | 60 ++++++++++++++++++++++++++-------- libgrive/src/base/Resource.hh | 2 +- libgrive/src/http/CurlAgent.cc | 1 - libgrive/src/http/Download.cc | 1 - libgrive/src/http/Header.cc | 8 +++++ libgrive/src/http/Header.hh | 1 + libgrive/src/util/Crypt.cc | 1 - libgrive/src/util/DateTime.cc | 1 - libgrive/src/util/Exception.cc | 1 - 11 files changed, 58 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 756873b8..2b0f69ef 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,7 @@ Enjoy! - Much faster and more correct synchronisation using local modification time and checksum cache (similar to git index) - Automatic move/rename detection, -m option removed - force option works again +- Instead of crashing on sync exceptions Grive will give a warning and attempt to sync failed files again during the next run. ### Grive2 v0.4.2 diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index 9dd80899..5e3eaaf3 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -35,7 +35,6 @@ #include #include #include -#include // for debugging only #include diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 66a4ac32..a0d571b6 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -28,6 +28,7 @@ #include "util/log/Log.hh" #include "util/OS.hh" #include "util/File.hh" +#include "http/Error.hh" #include #include @@ -264,18 +265,12 @@ void Resource::FromLocal( Val& state ) m_md5 = crypt::MD5::Get( path ); // File is changed locally. TODO: Detect conflicts is_changed = !state.Has( "md5" ) || m_md5 != state["md5"].Str(); - state.Set( "md5", Val( m_md5 ) ); } else is_changed = true; - state.Set( "ctime", Val( m_ctime.Sec() ) ); } if ( state.Has( "srv_time" ) ) m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; - if ( is_dir ) - state.Del( "md5" ); - else - state.Del( "tree" ); // follow parent recursively if ( m_parent->m_state == local_new || m_parent->m_state == remote_deleted ) @@ -418,6 +413,32 @@ void Resource::Sync( Syncer *syncer, ResourceTree *res_tree, const Val& options Log( "Error syncing %1%: %2%", Path(), e.what(), log::error ); return; } + catch ( http::Error &e ) + { + int *curlcode = boost::get_error_info< http::CurlCode > ( e ) ; + int *httpcode = boost::get_error_info< http::HttpResponseCode > ( e ) ; + std::string msg; + if ( curlcode ) + msg = *( boost::get_error_info< http::CurlErrMsg > ( e ) ); + else if ( httpcode ) + msg = "HTTP " + boost::to_string( *httpcode ); + else + msg = e.what(); + Log( "Error syncing %1%: %2%", Path(), msg, log::error ); + std::string *url = boost::get_error_info< http::Url > ( e ); + std::string *resp_hdr = boost::get_error_info< http::HttpResponseHeaders > ( e ); + std::string *resp_txt = boost::get_error_info< http::HttpResponseText > ( e ); + http::Header *req_hdr = boost::get_error_info< http::HttpRequestHeaders > ( e ); + if ( url ) + Log( "Request URL: %1%", *url, log::verbose ); + if ( req_hdr ) + Log( "Request headers: %1%", req_hdr->Str(), log::verbose ); + if ( resp_hdr ) + Log( "Response headers: %1%", *resp_hdr, log::verbose ); + if ( resp_txt ) + Log( "Response text: %1%", *resp_txt, log::verbose ); + return; + } // if myself is deleted, no need to do the childrens if ( m_state != local_deleted && m_state != remote_deleted ) @@ -463,11 +484,14 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti if ( syncer ) { if ( is_local ) + { syncer->Move( from, to->Parent(), to->Name() ); + to->SetIndex( false ); + } else { fs::rename( from->Path(), to->Path() ); - to->SetIndex(); + to->SetIndex( true ); } to->m_mtime = from->m_mtime; to->m_json->Set( "srv_time", Val( from->m_mtime.Sec() ) ); @@ -485,9 +509,11 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti case local_new : Log( "sync %1% doesn't exist in server, uploading", path, log::info ) ; - // FIXME: (?) do not write new timestamp on failed upload if ( syncer && syncer->Create( this ) ) + { m_state = sync ; + SetIndex( false ); + } break ; case local_deleted : @@ -502,7 +528,10 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti case local_changed : Log( "sync %1% changed in local. uploading", path, log::info ) ; if ( syncer && syncer->EditContent( this, options["new-rev"].Bool() ) ) + { m_state = sync ; + SetIndex( false ); + } break ; case remote_new : @@ -513,7 +542,7 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti fs::create_directories( path ) ; else syncer->Download( this, path ) ; - SetIndex() ; + SetIndex( true ) ; m_state = sync ; } break ; @@ -524,7 +553,7 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti if ( syncer ) { syncer->Download( this, path ) ; - SetIndex() ; + SetIndex( true ) ; m_state = sync ; } break ; @@ -604,21 +633,26 @@ void Resource::DeleteIndex() m_json = NULL; } -void Resource::SetIndex() +void Resource::SetIndex( bool re_stat ) { assert( m_parent->m_json != NULL ); if ( !m_json ) m_json = &((*m_parent->m_json)["tree"]).Item( Name() ); bool is_dir; - os::Stat( Path(), &m_ctime, NULL, &is_dir ); + if ( re_stat ) + os::Stat( Path(), &m_ctime, NULL, &is_dir ); if ( !is_dir ) { m_json->Set( "ctime", Val( m_ctime.Sec() ) ); m_json->Set( "md5", Val( m_md5 ) ); m_json->Del( "tree" ); } - else // check if tree item exists + else + { + // add tree item if it does not exist m_json->Item( "tree" ); + m_json->Del( "md5" ); + } } Resource::iterator Resource::begin() const diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index 354369fd..531b78d7 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -139,7 +139,7 @@ private : void DeleteLocal() ; void DeleteIndex() ; - void SetIndex() ; + void SetIndex( bool ) ; void SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& options ) ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index a527d519..4b23067e 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -35,7 +35,6 @@ #include #include #include -#include #include #include diff --git a/libgrive/src/http/Download.cc b/libgrive/src/http/Download.cc index 885187ef..26804e6f 100644 --- a/libgrive/src/http/Download.cc +++ b/libgrive/src/http/Download.cc @@ -20,7 +20,6 @@ #include "Download.hh" // #include "util/SignalHandler.hh" -#include "Error.hh" #include "util/Crypt.hh" // boost headers diff --git a/libgrive/src/http/Header.cc b/libgrive/src/http/Header.cc index 9f0bdd31..6d875f91 100644 --- a/libgrive/src/http/Header.cc +++ b/libgrive/src/http/Header.cc @@ -22,6 +22,7 @@ #include #include #include +#include namespace gr { namespace http { @@ -34,6 +35,13 @@ void Header::Add( const std::string& str ) m_vec.push_back( str ) ; } +std::string Header::Str() const +{ + std::ostringstream s ; + s << *this ; + return s.str() ; +} + Header::iterator Header::begin() const { return m_vec.begin() ; diff --git a/libgrive/src/http/Header.hh b/libgrive/src/http/Header.hh index 2486cd0c..844b6e6a 100644 --- a/libgrive/src/http/Header.hh +++ b/libgrive/src/http/Header.hh @@ -37,6 +37,7 @@ public : Header() ; void Add( const std::string& str ) ; + std::string Str() const ; iterator begin() const ; iterator end() const ; diff --git a/libgrive/src/util/Crypt.cc b/libgrive/src/util/Crypt.cc index 5fb8ebe8..b34494f9 100644 --- a/libgrive/src/util/Crypt.cc +++ b/libgrive/src/util/Crypt.cc @@ -24,7 +24,6 @@ #include "MemMap.hh" #include -#include // dependent libraries #include diff --git a/libgrive/src/util/DateTime.cc b/libgrive/src/util/DateTime.cc index ffb4ec57..2fe79891 100644 --- a/libgrive/src/util/DateTime.cc +++ b/libgrive/src/util/DateTime.cc @@ -33,7 +33,6 @@ #include #include #include -#include #include #include diff --git a/libgrive/src/util/Exception.cc b/libgrive/src/util/Exception.cc index 9cf66810..a4652f69 100644 --- a/libgrive/src/util/Exception.cc +++ b/libgrive/src/util/Exception.cc @@ -26,7 +26,6 @@ #include #include -#include namespace gr { From c76cdecad289d5db287972b489487e917d3a8012 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Tue, 5 Jan 2016 20:21:32 +0300 Subject: [PATCH 132/166] Note fixed revision support in readme --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2b0f69ef..0800e0ca 100644 --- a/README.md +++ b/README.md @@ -76,6 +76,7 @@ Enjoy! - Automatic move/rename detection, -m option removed - force option works again - Instead of crashing on sync exceptions Grive will give a warning and attempt to sync failed files again during the next run. +- Revision support works again. Grive 0.4.x always created new revisions for all files during sync, regardless of the absence of the --new-rev option. ### Grive2 v0.4.2 @@ -96,7 +97,7 @@ Known issues: First fork release, by Vitaliy Filippov / vitalif at mail*ru - Support for the new Google Drive REST API (old "Document List" API is shut down by Google 20 April 2015) -- REAL support for partial sync: syncs only one subdirectory with `grive -d subdir` +- REAL support for partial sync: syncs only one subdirectory with `grive -s subdir` - Major refactoring - a lot of dead code removed, JSON-C is not used anymore, API-specific code is split from non-API-specific - Some stability fixes from Visa Putkinen https://github.com/visap/grive/commits/visa - Slightly reduce number of syscalls when reading local files. From 60acb7596731364879a8252a1af16d62276a2f82 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 6 Jan 2016 01:37:52 +0300 Subject: [PATCH 133/166] Fix local index updating for items already in sync --- libgrive/src/base/Resource.cc | 12 +++++++----- libgrive/src/json/Val.cc | 2 ++ 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index a0d571b6..92708d06 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -574,14 +574,14 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti case sync : Log( "sync %1% already in sync", path, log::verbose ) ; + if ( !IsRoot() ) + SetIndex( false ) ; break ; // shouldn't go here case unknown : - assert( false ) ; - break ; - default : + assert( false ) ; break ; } @@ -635,15 +635,17 @@ void Resource::DeleteIndex() void Resource::SetIndex( bool re_stat ) { - assert( m_parent->m_json != NULL ); + assert( m_parent && m_parent->m_json != NULL ); if ( !m_json ) m_json = &((*m_parent->m_json)["tree"]).Item( Name() ); bool is_dir; if ( re_stat ) os::Stat( Path(), &m_ctime, NULL, &is_dir ); + else + is_dir = IsFolder(); + m_json->Set( "ctime", Val( m_ctime.Sec() ) ); if ( !is_dir ) { - m_json->Set( "ctime", Val( m_ctime.Sec() ) ); m_json->Set( "md5", Val( m_md5 ) ); m_json->Del( "tree" ); } diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index 0d8ca656..51ad62cc 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -116,6 +116,8 @@ const Val& Val::operator[]( std::size_t index ) const std::string Val::Str() const { + if ( Type() == int_type ) + return boost::to_string( As() ); return As() ; } From b6c0013052f837ba4bd4d1ed42bb3a55b72e78e0 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 6 Jan 2016 01:48:30 +0300 Subject: [PATCH 134/166] Fix subdir/subsubdir/newfile bug New files put in sub-subdirs of drive root were also trashed locally instead of begin uploaded to the server --- libgrive/src/base/Resource.cc | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 92708d06..1b888aef 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -272,14 +272,20 @@ void Resource::FromLocal( Val& state ) if ( state.Has( "srv_time" ) ) m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; - // follow parent recursively - if ( m_parent->m_state == local_new || m_parent->m_state == remote_deleted ) - m_state = m_parent->m_state ; - else + // Upload file if it is changed and remove if not. + // State will be updated to sync/remote_changed in FromRemote() + m_state = is_changed ? local_new : remote_deleted; + if ( m_state == local_new ) { - // Upload file if it is changed and remove if not. - // State will be updated to sync/remote_changed in FromRemote() - m_state = is_changed ? local_new : remote_deleted; + // local_new means this file is changed in local. + // this means we can't delete any of its parents. + // make sure their state is also set to local_new. + Resource *p = m_parent; + while ( p && p->m_state == remote_deleted ) + { + p->m_state = local_new; + p = p->m_parent; + } } } From 4a4e22026b42993777891897da2d4b9a548f77ab Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 6 Jan 2016 02:21:33 +0300 Subject: [PATCH 135/166] Fix json Val type check --- libgrive/src/json/Val.cc | 2 +- libgrive/src/json/Val.hh | 22 ++++++++-------------- libgrive/test/btest/ValTest.cc | 7 ++++--- 3 files changed, 13 insertions(+), 18 deletions(-) diff --git a/libgrive/src/json/Val.cc b/libgrive/src/json/Val.cc index 51ad62cc..2facfffd 100644 --- a/libgrive/src/json/Val.cc +++ b/libgrive/src/json/Val.cc @@ -123,7 +123,7 @@ std::string Val::Str() const Val::operator std::string() const { - return As() ; + return Str(); } int Val::Int() const diff --git a/libgrive/src/json/Val.hh b/libgrive/src/json/Val.hh index dfc2d331..4c8b9ccd 100644 --- a/libgrive/src/json/Val.hh +++ b/libgrive/src/json/Val.hh @@ -194,35 +194,29 @@ Val& Val::Assign( const T& t ) template const T& Val::As() const { - try - { - const Impl *impl = dynamic_cast *>( m_base.get() ) ; - return impl->val ; - } - catch ( std::exception& e ) + const Impl *impl = dynamic_cast *>( m_base.get() ) ; + if ( !impl ) { TypeEnum dest = Type2Enum::type ; BOOST_THROW_EXCEPTION( - Error() << SrcType_(Type()) << DestType_(dest) + Error() << SrcType_( Type() ) << DestType_( dest ) ) ; } + return impl->val ; } template T& Val::As() { - try - { - Impl *impl = dynamic_cast *>( m_base.get() ) ; - return impl->val ; - } - catch ( std::exception& e ) + Impl *impl = dynamic_cast *>( m_base.get() ) ; + if ( !impl ) { TypeEnum dest = Type2Enum::type ; BOOST_THROW_EXCEPTION( - Error() << SrcType_(Type()) << DestType_(dest) + Error() << SrcType_( Type() ) << DestType_( dest ) ) ; } + return impl->val ; } template diff --git a/libgrive/test/btest/ValTest.cc b/libgrive/test/btest/ValTest.cc index 5514825e..31c2cdb1 100644 --- a/libgrive/test/btest/ValTest.cc +++ b/libgrive/test/btest/ValTest.cc @@ -19,6 +19,7 @@ #include "json/Val.hh" #include +#include using namespace gr ; @@ -33,11 +34,11 @@ BOOST_FIXTURE_TEST_SUITE( ValTest, Fixture ) BOOST_AUTO_TEST_CASE( TestSimpleTypes ) { - Val null ; - BOOST_CHECK_EQUAL( null.Type(), Val::null_type ) ; - BOOST_CHECK( null.Is() ) ; + BOOST_CHECK_EQUAL( Val::Null().Type(), Val::null_type ) ; + BOOST_CHECK( Val::Null().Is() ) ; Val i( 100 ) ; + BOOST_CHECK_EQUAL( i.Str(), "100" ); BOOST_CHECK_EQUAL( i.As(), 100 ) ; BOOST_CHECK_EQUAL( i.Type(), Val::int_type ) ; } From 86acd18978c28cf95775ff0cce2623ba3964eb44 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 6 Jan 2016 12:55:02 +0300 Subject: [PATCH 136/166] Fix ResourceTest --- libgrive/src/base/Resource.cc | 3 ++- libgrive/test/base/ResourceTest.cc | 9 ++++++--- libgrive/test/data/test_dir1.state | 1 - 3 files changed, 8 insertions(+), 5 deletions(-) delete mode 100644 libgrive/test/data/test_dir1.state diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 1b888aef..8fc67a94 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -164,6 +164,7 @@ void Resource::FromRemoteFile( const Entry& remote ) } // local not exists + // FIXME: Remove additional stat() call here else if ( !fs::exists( path ) ) { Trace( "file %1% change stamp = %2%", Path(), remote.ChangeStamp() ) ; @@ -704,7 +705,7 @@ std::string Resource::MD5() const bool Resource::IsRoot() const { // Root entry does not show up in file feeds, so we check for empty parent (and self-href) - return m_parent == 0 ; + return !m_parent ; } bool Resource::HasID() const diff --git a/libgrive/test/base/ResourceTest.cc b/libgrive/test/base/ResourceTest.cc index cbbe6c86..f40dc38b 100644 --- a/libgrive/test/base/ResourceTest.cc +++ b/libgrive/test/base/ResourceTest.cc @@ -40,8 +40,8 @@ ResourceTest::ResourceTest( ) void ResourceTest::TestRootPath() { - std::string rootFolder = "/home/usr/grive/grive"; - Resource root(rootFolder) ; + std::string rootFolder = "/home/usr/grive/grive"; + Resource root( rootFolder ) ; CPPUNIT_ASSERT( root.IsRoot() ) ; GRUT_ASSERT_EQUAL( root.Path(), fs::path( rootFolder ) ) ; } @@ -52,20 +52,23 @@ void ResourceTest::TestNormal( ) Resource subject( "entry.xml", "file" ) ; root.AddChild( &subject ) ; + GRUT_ASSERT_EQUAL( subject.IsRoot(), false ) ; GRUT_ASSERT_EQUAL( subject.Path(), fs::path( TEST_DATA ) / "entry.xml" ) ; Val st; + st.Add( "srv_time", Val( DateTime( "2012-05-09T16:13:22.401Z" ).Sec() ) ); subject.FromLocal( st ) ; GRUT_ASSERT_EQUAL( subject.MD5(), "c0742c0a32b2c909b6f176d17a6992d0" ) ; GRUT_ASSERT_EQUAL( subject.StateStr(), "local_new" ) ; xml::Node entry = xml::Node::Element( "entry" ) ; entry.AddElement( "updated" ).AddText( "2012-05-09T16:13:22.401Z" ) ; + entry.AddElement( "docs:md5Checksum" ).AddText( "DIFFERENT" ) ; Entry1 remote( entry ) ; + GRUT_ASSERT_EQUAL( "different", remote.MD5() ) ; subject.FromRemote( remote ) ; GRUT_ASSERT_EQUAL( "local_changed", subject.StateStr() ) ; } - } // end of namespace grut diff --git a/libgrive/test/data/test_dir1.state b/libgrive/test/data/test_dir1.state deleted file mode 100644 index 67b5d1b9..00000000 --- a/libgrive/test/data/test_dir1.state +++ /dev/null @@ -1 +0,0 @@ -{ "change_stamp": "", "rtree": { "name": ".", "id": "folder:root", "href": "https:\/\/docs.google.com\/feeds\/default\/private\/full\/folder%3Aroot", "md5": "", "kind": "folder", "mtime": { "sec": 0, "nsec": 0 }, "child": [ { "name": "entry.xml", "id": "", "href": "", "md5": "c0742c0a32b2c909b6f176d17a6992d0", "kind": "file", "mtime": { "sec": 1336796872, "nsec": 404985662 }, "child": [ ] } ] } } \ No newline at end of file From 98416354f75dc9244e75da2740f48d2ebc9f28a6 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Mon, 11 Jan 2016 18:15:29 +0300 Subject: [PATCH 137/166] Remove "me in readers" condition (fix #37) --- libgrive/src/drive2/Syncer2.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index ae6bc4d4..fda18e4b 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -213,12 +213,12 @@ bool Syncer2::Upload( Resource *res, bool new_rev ) std::auto_ptr Syncer2::GetFolders() { - return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers+and+trashed%3dfalse+and+mimeType%3d%27" + mime_types::folder + "%27" ) ); + return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=trashed%3dfalse+and+mimeType%3d%27" + mime_types::folder + "%27" ) ); } std::auto_ptr Syncer2::GetAll() { - return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=%27me%27+in+readers+and+trashed%3dfalse" ) ); + return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=trashed%3dfalse" ) ); } std::string ChangesFeed( long changestamp, int maxResults = 1000 ) From ae06eccb38b2fe250c9ddeac3e3973f80b8a0aa9 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 14 Jan 2016 23:46:26 +0300 Subject: [PATCH 138/166] Release 0.5 --- CMakeLists.txt | 2 +- README.md | 9 +++++---- debian/changelog | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index fac9c1af..e1be366a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! -set( GRIVE_VERSION "0.5-pre" ) +set( GRIVE_VERSION "0.5" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) diff --git a/README.md b/README.md index 0800e0ca..08f07354 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# Grive2 0.5-dev +# Grive2 0.5 -3 Jan 2016, Vitaliy Filippov +14 Jan 2016, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 @@ -16,7 +16,7 @@ directory named .trash or put them in the Google Drive trash. You can always rec There are a few things that Grive does not do at the moment: - continously wait for changes in file system or in Google Drive to occur and upload. - A sync is only performed when you run Grive, and it calculates checksums for all files every time. + A sync is only performed when you run Grive. - symbolic links support. - support for Google documents. @@ -70,13 +70,14 @@ Enjoy! ## Version History -### Grive2 v0.5 (unreleased) +### Grive2 v0.5 - Much faster and more correct synchronisation using local modification time and checksum cache (similar to git index) - Automatic move/rename detection, -m option removed - force option works again - Instead of crashing on sync exceptions Grive will give a warning and attempt to sync failed files again during the next run. - Revision support works again. Grive 0.4.x always created new revisions for all files during sync, regardless of the absence of the --new-rev option. +- Shared files now sync correctly ### Grive2 v0.4.2 diff --git a/debian/changelog b/debian/changelog index 39d29bda..ff76816a 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,4 +1,4 @@ -grive2 (0.5+git20160103) unstable; urgency=medium +grive2 (0.5+git20160114) unstable; urgency=medium * Newer release, with support for faster sync and rename detection From cfb8ff08b300d1fa57e12cdc85f724e3e25c906d Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 14 Jan 2016 23:47:35 +0300 Subject: [PATCH 139/166] Move to 0.5.1-dev --- CMakeLists.txt | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index e1be366a..7b330d65 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 2.8) # Grive version. remember to update it for every new release! -set( GRIVE_VERSION "0.5" ) +set( GRIVE_VERSION "0.5.1-dev" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) diff --git a/README.md b/README.md index 08f07354..ce32d50c 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Grive2 0.5 +# Grive2 0.5.1-dev 14 Jan 2016, Vitaliy Filippov From 40e33cb524a17586c392a24d4f4ea4a8e33f693b Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Tue, 10 May 2016 00:26:10 +0300 Subject: [PATCH 140/166] Implement upload-only and no-remote-new modes (fix #69) --- README.md | 4 ++++ grive/doc/grive.1 | 10 ++++++++++ grive/src/main.cc | 2 ++ libgrive/src/base/Resource.cc | 36 ++++++++++++++++++++++------------- libgrive/src/util/Config.cc | 4 +++- 5 files changed, 42 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index ce32d50c..afde8e12 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,10 @@ Enjoy! ## Version History +### Grive2 v0.5.1-dev + +- no-remote-new and upload-only modes + ### Grive2 v0.5 - Much faster and more correct synchronisation using local modification time and checksum cache (similar to git index) diff --git a/grive/doc/grive.1 b/grive/doc/grive.1 index 96efbff3..ad9ea0d3 100644 --- a/grive/doc/grive.1 +++ b/grive/doc/grive.1 @@ -33,6 +33,16 @@ Forces .I grive to always download a file from Google Drive instead uploading it .TP +\fB\-u, \-\-upload\-only\fR +Forces +.I grive +to not download anything from Google Drive and only upload local changes to server instead +.TP +\fB\-n, \-\-no\-remote\-new\fR +Forces +.I grive +to download only files that are changed in Google Drive and already exist locally +.TP \fB\-h\fR, \fB\-\-help\fR Produces help message .TP diff --git a/grive/src/main.cc b/grive/src/main.cc index b0fb4ae8..60383136 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -118,6 +118,8 @@ int Main( int argc, char **argv ) ( "log,l", po::value(), "Set log output filename." ) ( "force,f", "Force grive to always download a file from Google Drive " "instead of uploading it." ) + ( "upload-only,u", "Do not download anything from Google Drive, only upload local changes" ) + ( "no-remote-new,n", "Download only files that are changed in Google Drive and already exist locally" ) ( "dry-run", "Only detect which files need to be uploaded/downloaded, " "without actually performing them." ) ( "ignore", po::value(), "Perl RegExp to ignore files (matched against relative paths, remembered for next runs)." ) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 8fc67a94..9cc441e2 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -542,26 +542,36 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti break ; case remote_new : - Log( "sync %1% created in remote. creating local", path, log::info ) ; - if ( syncer ) + if ( options["no-remote-new"].Bool() ) + Log( "sync %1% created in remote. skipping", path, log::info ) ; + else { - if ( IsFolder() ) - fs::create_directories( path ) ; - else - syncer->Download( this, path ) ; - SetIndex( true ) ; - m_state = sync ; + Log( "sync %1% created in remote. creating local", path, log::info ) ; + if ( syncer ) + { + if ( IsFolder() ) + fs::create_directories( path ) ; + else + syncer->Download( this, path ) ; + SetIndex( true ) ; + m_state = sync ; + } } break ; case remote_changed : assert( !IsFolder() ) ; - Log( "sync %1% changed in remote. downloading", path, log::info ) ; - if ( syncer ) + if ( options["upload-only"].Bool() ) + Log( "sync %1% changed in remote. skipping", path, log::info ) ; + else { - syncer->Download( this, path ) ; - SetIndex( true ) ; - m_state = sync ; + Log( "sync %1% changed in remote. downloading", path, log::info ) ; + if ( syncer ) + { + syncer->Download( this, path ) ; + SetIndex( true ) ; + m_state = sync ; + } } break ; diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index b8483c82..4d03108e 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -46,8 +46,10 @@ Config::Config( const po::variables_map& vm ) m_cmd.Add( "dir", Val(vm.count("dir") > 0 ? vm["dir"].as() : "" ) ) ; - if ( vm.count("ignore") > 0 ) + if ( vm.count( "ignore" ) > 0 ) m_cmd.Add( "ignore", Val( vm["ignore"].as() ) ); + m_cmd.Add( "no-remote-new", Val( vm.count( "no-remote-new" ) > 0 || vm.count( "upload-only" ) > 0 ) ); + m_cmd.Add( "upload-only", Val( vm.count( "upload-only" ) > 0 ) ); m_path = GetPath( fs::path(m_cmd["path"].Str()) ) ; m_file = Read( ) ; From 62e26118f21fd3fc7a2e6289dc68792404e8a92b Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Tue, 10 May 2016 23:10:41 +0300 Subject: [PATCH 141/166] Fix #62 - sub-subdirectory sync with -s --- libgrive/src/base/State.cc | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 83232167..c2b3865f 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -51,7 +51,14 @@ State::State( const fs::path& filename, const Val& options ) : { // "-s" is internally converted to an ignore regexp const boost::regex esc( "[.^$|()\\[\\]{}*+?\\\\]" ); - std::string ign = "^(?!"+regex_replace( m_dir, esc, "\\\\&", boost::format_sed )+"(/|$))"; + m_dir = regex_replace( m_dir, esc, "\\\\&", boost::format_sed ); + size_t pos = 0; + while ( ( pos = m_dir.find( '/', pos ) ) != std::string::npos ) + { + m_dir = m_dir.substr( 0, pos ) + "$|" + m_dir; + pos = pos*2 + 3; + } + std::string ign = "^(?!"+m_dir+"(/|$))"; m_ign = ign; } } From 5327016d36f34f9dabb8d5357ee4ff82e2d0832b Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 11 May 2016 15:56:59 +0300 Subject: [PATCH 142/166] Make ignore regexp non-persistent (fix #58) --- README.md | 2 ++ libgrive/src/base/Resource.cc | 2 +- libgrive/src/base/State.cc | 31 ++++++++++++++++++++----------- libgrive/src/base/State.hh | 1 + libgrive/src/util/Config.cc | 1 + 5 files changed, 25 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index afde8e12..e5ec3e92 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,8 @@ Enjoy! ### Grive2 v0.5.1-dev - no-remote-new and upload-only modes +- ignore regexp does not persist anymore (note that Grive will still track it to not + accidentally delete remote files when changing ignore regexp) ### Grive2 v0.5 diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 9cc441e2..003e8940 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -525,7 +525,7 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti case local_deleted : Log( "sync %1% deleted in local. deleting remote", path, log::info ) ; - if ( syncer ) + if ( syncer && !options["no-delete-remote"].Bool() ) { syncer->DeleteRemote( this ) ; DeleteIndex() ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index c2b3865f..44d8bf91 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -41,6 +41,8 @@ State::State( const fs::path& filename, const Val& options ) : // the "-f" option will make grive always think remote is newer m_force = options.Has( "force" ) ? options["force"].Bool() : false ; + std::string m_orig_ign = m_ign; + m_ign = ""; if ( options.Has( "ignore" ) && options["ignore"].Str() != m_ign ) m_ign = options["ignore"].Str(); else if ( options.Has( "dir" ) ) @@ -62,7 +64,8 @@ State::State( const fs::path& filename, const Val& options ) : m_ign = ign; } } - + + m_ign_changed = m_orig_ign != "" && m_orig_ign != m_ign; m_ign_re = boost::regex( m_ign.empty() ? "^\\.(grive|grive_state|trash)" : ( m_ign+"|^\\.(grive|grive_state|trash)" ) ); } @@ -120,18 +123,24 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) for( Val::Object::iterator i = leftover.begin(); i != leftover.end(); i++ ) { - // Restore state of locally deleted files - Resource *c = folder->FindChild( i->first ) ; - if ( !c ) + std::string path = folder->IsRoot() ? i->first : ( folder->RelPath() / i->first ).string(); + if ( IsIgnore( path ) ) + Log( "file %1% is ignored by grive", path, log::verbose ) ; + else { - c = new Resource( i->first, i->second.Has( "tree" ) ? "folder" : "file" ) ; - folder->AddChild( c ) ; - m_res.Insert( c ) ; + // Restore state of locally deleted files + Resource *c = folder->FindChild( i->first ) ; + if ( !c ) + { + c = new Resource( i->first, i->second.Has( "tree" ) ? "folder" : "file" ) ; + folder->AddChild( c ) ; + m_res.Insert( c ) ; + } + Val& rec = tree.Item( i->first ); + if ( m_force || m_ign_changed ) + rec.Del( "srv_time" ); + c->FromDeleted( rec ); } - Val& rec = tree.Item( i->first ); - if ( m_force ) - rec.Del( "srv_time" ); - c->FromDeleted( rec ); } } diff --git a/libgrive/src/base/State.hh b/libgrive/src/base/State.hh index 088be382..68b6df88 100644 --- a/libgrive/src/base/State.hh +++ b/libgrive/src/base/State.hh @@ -78,6 +78,7 @@ private : boost::regex m_ign_re ; Val m_st ; bool m_force ; + bool m_ign_changed ; std::list m_unresolved ; } ; diff --git a/libgrive/src/util/Config.cc b/libgrive/src/util/Config.cc index 4d03108e..ba0c91e3 100644 --- a/libgrive/src/util/Config.cc +++ b/libgrive/src/util/Config.cc @@ -50,6 +50,7 @@ Config::Config( const po::variables_map& vm ) m_cmd.Add( "ignore", Val( vm["ignore"].as() ) ); m_cmd.Add( "no-remote-new", Val( vm.count( "no-remote-new" ) > 0 || vm.count( "upload-only" ) > 0 ) ); m_cmd.Add( "upload-only", Val( vm.count( "upload-only" ) > 0 ) ); + m_cmd.Add( "no-delete-remote", Val( vm.count( "no-delete-remote" ) > 0 ) ); m_path = GetPath( fs::path(m_cmd["path"].Str()) ) ; m_file = Read( ) ; From 44d3ddf9287a61fa60501fb55911f314c13d882f Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 15 May 2016 23:05:25 +0300 Subject: [PATCH 143/166] Implement upload/download speed limit (fix #60) --- README.md | 1 + grive/src/main.cc | 7 +++++++ libgrive/src/http/Agent.cc | 10 ++++++++++ libgrive/src/http/Agent.hh | 6 ++++++ libgrive/src/http/CurlAgent.cc | 4 ++++ libgrive/src/protocol/AuthAgent.cc | 10 ++++++++++ libgrive/src/protocol/AuthAgent.hh | 3 +++ 7 files changed, 41 insertions(+) diff --git a/README.md b/README.md index e5ec3e92..d6407c64 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,7 @@ Enjoy! - no-remote-new and upload-only modes - ignore regexp does not persist anymore (note that Grive will still track it to not accidentally delete remote files when changing ignore regexp) +- added options to limit upload and download speed ### Grive2 v0.5 diff --git a/grive/src/main.cc b/grive/src/main.cc index 60383136..281245b5 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -123,6 +123,8 @@ int Main( int argc, char **argv ) ( "dry-run", "Only detect which files need to be uploaded/downloaded, " "without actually performing them." ) ( "ignore", po::value(), "Perl RegExp to ignore files (matched against relative paths, remembered for next runs)." ) + ( "upload-speed,U", po::value(), "Limit upload speed in kbytes per second" ) + ( "download-speed,D", po::value(), "Limit download speed in kbytes per second" ) ; po::variables_map vm; @@ -195,6 +197,11 @@ int Main( int argc, char **argv ) AuthAgent agent( token, http.get() ) ; v2::Syncer2 syncer( &agent ); + if ( vm.count( "upload-speed" ) > 0 ) + agent.SetUploadSpeed( vm["upload-speed"].as() * 1000 ); + if ( vm.count( "download-speed" ) > 0 ) + agent.SetDownloadSpeed( vm["download-speed"].as() * 1000 ); + Drive drive( &syncer, config.GetAll() ) ; drive.DetectChanges() ; diff --git a/libgrive/src/http/Agent.cc b/libgrive/src/http/Agent.cc index b9143c65..ba048468 100644 --- a/libgrive/src/http/Agent.cc +++ b/libgrive/src/http/Agent.cc @@ -64,4 +64,14 @@ long Agent::Post( return Request( "POST", url, &s, dest, h ); } +void Agent::SetUploadSpeed( unsigned kbytes ) +{ + mMaxUpload = kbytes; +} + +void Agent::SetDownloadSpeed( unsigned kbytes ) +{ + mMaxDownload = kbytes; +} + } } // end of namespace diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index cb8594af..ce1883b8 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -34,6 +34,9 @@ class Header ; class Agent { +protected: + unsigned mMaxUpload, mMaxDownload ; + public : virtual ~Agent() {} @@ -70,6 +73,9 @@ public : DataStream *dest, const Header& hdr ) = 0 ; + virtual void SetUploadSpeed( unsigned kbytes ) ; + virtual void SetDownloadSpeed( unsigned kbytes ) ; + virtual std::string LastError() const = 0 ; virtual std::string LastErrorHeaders() const = 0 ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 4b23067e..3a279550 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -86,6 +86,10 @@ void CurlAgent::Init() ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADERFUNCTION, &CurlAgent::HeaderCallback ) ; ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADERDATA, this ) ; ::curl_easy_setopt( m_pimpl->curl, CURLOPT_HEADER, 0L ) ; + if ( mMaxUpload > 0 ) + ::curl_easy_setopt( m_pimpl->curl, CURLOPT_MAX_SEND_SPEED_LARGE, mMaxUpload ) ; + if ( mMaxDownload > 0 ) + ::curl_easy_setopt( m_pimpl->curl, CURLOPT_MAX_RECV_SPEED_LARGE, mMaxDownload ) ; m_pimpl->error = false; m_pimpl->error_headers = ""; m_pimpl->error_data = ""; diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 77cb54f0..65665f70 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -47,6 +47,16 @@ void AuthAgent::SetLog( http::ResponseLog *log ) return m_agent->SetLog( log ); } +void AuthAgent::SetUploadSpeed( unsigned kbytes ) +{ + m_agent->SetUploadSpeed( kbytes ); +} + +void AuthAgent::SetDownloadSpeed( unsigned kbytes ) +{ + m_agent->SetDownloadSpeed( kbytes ); +} + http::Header AuthAgent::AppendHeader( const http::Header& hdr ) const { http::Header h(hdr) ; diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index 3dd3402a..7f16dc14 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -54,6 +54,9 @@ public : std::string Escape( const std::string& str ) ; std::string Unescape( const std::string& str ) ; + void SetUploadSpeed( unsigned kbytes ) ; + void SetDownloadSpeed( unsigned kbytes ) ; + private : http::Header AppendHeader( const http::Header& hdr ) const ; bool CheckRetry( long response ) ; From 1bd86307c679f5fd6c90868ebb5b146e19d5177c Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Mon, 16 May 2016 12:29:29 +0300 Subject: [PATCH 144/166] Allow 5 params in Log, fix typo --- libgrive/src/base/Resource.cc | 2 +- libgrive/src/util/log/Log.hh | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 003e8940..fecbe56c 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -186,7 +186,7 @@ void Resource::FromRemoteFile( const Entry& remote ) // remote checksum unknown, assume the file is not changed in remote else if ( remote.MD5().empty() ) { - Log( "file %1% has unknown checksum in remote. assuned in sync", + Log( "file %1% has unknown checksum in remote. assumed in sync", Path(), log::verbose ) ; m_state = sync ; } diff --git a/libgrive/src/util/log/Log.hh b/libgrive/src/util/log/Log.hh index 9646042f..762320c3 100644 --- a/libgrive/src/util/log/Log.hh +++ b/libgrive/src/util/log/Log.hh @@ -115,6 +115,12 @@ void Log( LogBase::Inst()->Log( log::Fmt(fmt) % p1 % p2 % p3 % p4, s ) ; } +template +void Log( const std::string& fmt, const P1& p1, const P2& p2, const P3& p3, const P4& p4, const P5& p5, log::Serverity s = log::info ) +{ + LogBase::Inst()->Log( log::Fmt(fmt) % p1 % p2 % p3 % p4 % p5, s ) ; +} + void Trace( const std::string& str ) ; template From 46dfa1abfaf5bc35f7400e618467504fd2c79013 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 21 May 2016 12:45:43 +0300 Subject: [PATCH 145/166] Add boost_INCLUDE_DIRS (fix #77) --- libgrive/CMakeLists.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 90ef451d..39d64f72 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -44,6 +44,7 @@ endif ( ZLIB_FOUND ) include_directories( ${libgrive_SOURCE_DIR}/src ${libgrive_SOURCE_DIR}/test + ${Boost_INCLUDE_DIRS} ${OPT_INCS} ${YAJL_INCLUDE_DIRS} ) From 76827a760c8220ac8543e8834ba94875b6078e09 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 30 Jun 2016 13:16:46 +0300 Subject: [PATCH 146/166] Allow libstdc++-6-dev --- debian/control | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/control b/debian/control index 2b658956..d2e09e37 100644 --- a/debian/control +++ b/debian/control @@ -2,7 +2,7 @@ Source: grive2 Section: net Priority: optional Maintainer: Vitaliy Filippov -Build-Depends: debhelper, cmake, pkg-config, zlib1g-dev, libcurl4-openssl-dev, libstdc++6-4.4-dev | libstdc++-4.9-dev | libstdc++-5-dev, libboost-filesystem-dev, libboost-program-options-dev, libboost-test-dev, libboost-regex-dev, libexpat1-dev, binutils-dev, libgcrypt-dev, libyajl-dev +Build-Depends: debhelper, cmake, pkg-config, zlib1g-dev, libcurl4-openssl-dev, libstdc++-6-dev | libstdc++6-4.4-dev | libstdc++-4.9-dev | libstdc++-5-dev, libboost-filesystem-dev, libboost-program-options-dev, libboost-test-dev, libboost-regex-dev, libexpat1-dev, binutils-dev, libgcrypt-dev, libyajl-dev Standards-Version: 3.9.6 Homepage: https://yourcmc.ru/wiki/Grive2 From 2727160257a52213586e710c4782735627fd80dc Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 30 Jun 2016 13:18:25 +0300 Subject: [PATCH 147/166] Note that ignore regexp and subdir are not persistent anymore (sorry :)) --- grive/src/main.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index 281245b5..477825ff 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -110,7 +110,7 @@ int Main( int argc, char **argv ) ( "version,v", "Display Grive version" ) ( "auth,a", "Request authorization token" ) ( "path,p", po::value(), "Root directory to sync") - ( "dir,s", po::value(), "Single subdirectory to sync (remembered for next runs)") + ( "dir,s", po::value(), "Single subdirectory to sync") ( "verbose,V", "Verbose mode. Enable more messages than normal.") ( "log-http", po::value(), "Log all HTTP responses in this file for debugging.") ( "new-rev", "Create new revisions in server for updated files.") @@ -122,7 +122,7 @@ int Main( int argc, char **argv ) ( "no-remote-new,n", "Download only files that are changed in Google Drive and already exist locally" ) ( "dry-run", "Only detect which files need to be uploaded/downloaded, " "without actually performing them." ) - ( "ignore", po::value(), "Perl RegExp to ignore files (matched against relative paths, remembered for next runs)." ) + ( "ignore", po::value(), "Perl RegExp to ignore files (matched against relative paths)." ) ( "upload-speed,U", po::value(), "Limit upload speed in kbytes per second" ) ( "download-speed,D", po::value(), "Limit download speed in kbytes per second" ) ; From fbf8f1663fe768b45553f337e9bcea26124b0aa3 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 6 Jul 2016 12:04:15 +0300 Subject: [PATCH 148/166] Initialize mMaxUpload/mMaxDownload to 0 (fix #81) --- libgrive/src/http/Agent.cc | 5 +++++ libgrive/src/http/Agent.hh | 1 + libgrive/src/http/CurlAgent.cc | 2 +- libgrive/src/protocol/AuthAgent.cc | 1 + 4 files changed, 8 insertions(+), 1 deletion(-) diff --git a/libgrive/src/http/Agent.cc b/libgrive/src/http/Agent.cc index ba048468..39276ac0 100644 --- a/libgrive/src/http/Agent.cc +++ b/libgrive/src/http/Agent.cc @@ -25,6 +25,11 @@ namespace gr { namespace http { +Agent::Agent() +{ + mMaxUpload = mMaxDownload = 0; +} + long Agent::Put( const std::string& url, const std::string& data, diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index ce1883b8..8268e197 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -38,6 +38,7 @@ protected: unsigned mMaxUpload, mMaxDownload ; public : + Agent() ; virtual ~Agent() {} virtual ResponseLog* GetLog() const = 0 ; diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 3a279550..3bfdfca9 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -72,7 +72,7 @@ struct CurlAgent::Impl static struct curl_slist* SetHeader( CURL* handle, const Header& hdr ); -CurlAgent::CurlAgent() : +CurlAgent::CurlAgent() : Agent(), m_pimpl( new Impl ) { m_pimpl->curl = ::curl_easy_init(); diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index 65665f70..f2f59d6f 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -32,6 +32,7 @@ namespace gr { using namespace http ; AuthAgent::AuthAgent( OAuth2& auth, Agent *real_agent ) : + Agent(), m_auth ( auth ), m_agent ( real_agent ) { From 457d84974592ed3f123622b12f2a7a748cbc0668 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 13 Jul 2016 21:13:52 +0300 Subject: [PATCH 149/166] Update package files --- debian/changelog | 6 ++++++ debian/control | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/debian/changelog b/debian/changelog index ff76816a..6b2790ab 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +grive2 (0.5.1+git20160706) unstable; urgency=medium + + * Newer dev version + + -- Vitaliy Filippov Wed, 13 Jul 2016 21:12:55 +0300 + grive2 (0.5+git20160114) unstable; urgency=medium * Newer release, with support for faster sync and rename detection diff --git a/debian/control b/debian/control index d2e09e37..edad2557 100644 --- a/debian/control +++ b/debian/control @@ -2,7 +2,7 @@ Source: grive2 Section: net Priority: optional Maintainer: Vitaliy Filippov -Build-Depends: debhelper, cmake, pkg-config, zlib1g-dev, libcurl4-openssl-dev, libstdc++-6-dev | libstdc++6-4.4-dev | libstdc++-4.9-dev | libstdc++-5-dev, libboost-filesystem-dev, libboost-program-options-dev, libboost-test-dev, libboost-regex-dev, libexpat1-dev, binutils-dev, libgcrypt-dev, libyajl-dev +Build-Depends: debhelper, cmake, pkg-config, zlib1g-dev, libcurl4-openssl-dev | libcurl4-gnutls-dev, libstdc++-6-dev | libstdc++6-4.4-dev | libstdc++-4.9-dev | libstdc++-5-dev, libboost-filesystem-dev, libboost-program-options-dev, libboost-test-dev, libboost-regex-dev, libexpat1-dev, binutils-dev, libgcrypt-dev, libyajl-dev Standards-Version: 3.9.6 Homepage: https://yourcmc.ru/wiki/Grive2 From e91a2b598b74a505729e96c070a6fd9cc39c967d Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 27 Jul 2016 17:21:28 +0300 Subject: [PATCH 150/166] Name it working copy, not just root (fixes #86) --- grive/doc/grive.1 | 7 ++++--- grive/src/main.cc | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/grive/doc/grive.1 b/grive/doc/grive.1 index ad9ea0d3..3b0b56ad 100644 --- a/grive/doc/grive.1 +++ b/grive/doc/grive.1 @@ -62,9 +62,10 @@ for debugging \fB\-\-new\-rev\fR Create new revisions in server for updated files .TP -\fB\-p\fR , \fB\-\-path\fR -Set root sync directory to -.I +\fB\-p\fR , \fB\-\-path\fR +Use +.I +as the working copy root directory .TP \fB\-s\fR , \fB\-\-dir\fR Sync a single diff --git a/grive/src/main.cc b/grive/src/main.cc index 477825ff..0894545b 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -109,7 +109,7 @@ int Main( int argc, char **argv ) ( "help,h", "Produce help message" ) ( "version,v", "Display Grive version" ) ( "auth,a", "Request authorization token" ) - ( "path,p", po::value(), "Root directory to sync") + ( "path,p", po::value(), "Path to working copy root") ( "dir,s", po::value(), "Single subdirectory to sync") ( "verbose,V", "Verbose mode. Enable more messages than normal.") ( "log-http", po::value(), "Log all HTTP responses in this file for debugging.") From a756414e7164ed193c422b698b4d4c8f5e682ae5 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 31 Jul 2016 22:05:03 +0300 Subject: [PATCH 151/166] Skip unreadable files during sync, remove extra stat calls (fix #63) --- debian/changelog | 4 ++-- libgrive/src/base/Resource.cc | 38 ++++++++++++++++++++++++++++------- libgrive/src/base/Resource.hh | 1 + 3 files changed, 34 insertions(+), 9 deletions(-) diff --git a/debian/changelog b/debian/changelog index 6b2790ab..db47f06e 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,8 +1,8 @@ -grive2 (0.5.1+git20160706) unstable; urgency=medium +grive2 (0.5.1+git20160731) unstable; urgency=medium * Newer dev version - -- Vitaliy Filippov Wed, 13 Jul 2016 21:12:55 +0300 + -- Vitaliy Filippov Wed, 31 Jul 2016 22:04:53 +0300 grive2 (0.5+git20160114) unstable; urgency=medium diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index fecbe56c..72c3a267 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -52,7 +52,8 @@ Resource::Resource( const fs::path& root_folder ) : m_is_editable( true ), m_parent ( 0 ), m_state ( sync ), - m_json ( NULL ) + m_json ( NULL ), + m_local_exists( true ) { } @@ -62,7 +63,8 @@ Resource::Resource( const std::string& name, const std::string& kind ) : m_is_editable( true ), m_parent ( 0 ), m_state ( unknown ), - m_json ( NULL ) + m_json ( NULL ), + m_local_exists( false ) { } @@ -87,17 +89,22 @@ void Resource::FromRemoteFolder( const Entry& remote ) Log( "folder %1% is read-only", path, log::verbose ) ; // already sync - if ( fs::is_directory( path ) ) + if ( m_local_exists && m_kind == "folder" ) { Log( "folder %1% is in sync", path, log::verbose ) ; m_state = sync ; } - else if ( fs::exists( path ) ) + else if ( m_local_exists && m_kind == "file" ) { // TODO: handle type change Log( "%1% changed from folder to file", path, log::verbose ) ; m_state = sync ; } + else if ( m_local_exists && m_kind == "bad" ) + { + Log( "%1% inaccessible", path, log::verbose ) ; + m_state = sync ; + } else if ( remote.MTime().Sec() > m_mtime.Sec() ) // FIXME only seconds are stored in local index { // remote folder created after last sync, so remote is newer @@ -163,9 +170,13 @@ void Resource::FromRemoteFile( const Entry& remote ) m_state = m_parent->m_state ; } + else if ( m_kind == "bad" ) + { + m_state = sync; + } + // local not exists - // FIXME: Remove additional stat() call here - else if ( !fs::exists( path ) ) + else if ( !m_local_exists ) { Trace( "file %1% change stamp = %2%", Path(), remote.ChangeStamp() ) ; @@ -246,10 +257,23 @@ void Resource::FromLocal( Val& state ) { fs::path path = Path() ; bool is_dir; - os::Stat( path, &m_ctime, NULL, &is_dir ) ; + try + { + os::Stat( path, &m_ctime, NULL, &is_dir ) ; + } + catch ( os::Error &e ) + { + // invalid symlink, unreadable file or something else + int const* eno = boost::get_error_info< boost::errinfo_errno >(e); + Log( "Error accessing %1%: %2%; skipping file", path.string(), strerror( *eno ), log::warning ); + m_state = sync; + m_kind = "bad"; + return; + } m_name = path.filename().string() ; m_kind = is_dir ? "folder" : "file"; + m_local_exists = true; bool is_changed; if ( state.Has( "ctime" ) && (u64_t) m_ctime.Sec() <= state["ctime"].U64() && diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index 531b78d7..e7ee0ac3 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -162,6 +162,7 @@ private : State m_state ; Val* m_json ; + bool m_local_exists ; } ; } // end of namespace gr::v1 From 195e5091c73e6ec6fac3682aa60c3d7dee0f27c2 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Mon, 29 Aug 2016 13:29:12 +0300 Subject: [PATCH 152/166] Add debian package note to readme --- README.md | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index d6407c64..9cbc4e83 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,24 @@ There are a few things that Grive does not do at the moment: These may be added in the future, possibly the next release. +When Grive is run for the first time, you should use the "-a" argument to grant +permission to Grive to access to your Google Drive. A URL should be printed. +Go to the link. You will need to login to your Google account if you haven't +done so. After granting the permission to Grive, the browser will show you +an authenication code. Copy-and-paste that to the standard input of Grive. + +If everything works fine, Grive will create .grive and .grive_state files in your +current directory. It will also start downloading files from your Google Drive to +your current directory. + +Enjoy! + +## Installation + +For the detailed instructions, see http://yourcmc.ru/wiki/Grive2#Installation + +### Install dependencies + You need the following libraries: - yajl 2.x @@ -46,6 +64,15 @@ FreeBSD: pkg install git cmake boost-libs yajl libgcrypt pkgconf cppunit libbfd +### Build Debian packages + +On a Debian/Ubuntu/Linux Mint you can use `dpkg-buildpackage` utility from `dpkg-dev` package +to build grive. Just clone the repository, `cd` into it and run + + dpkg-buildpackage -j4 + +### Manual build + Grive uses cmake to build. Basic install sequence is mkdir build @@ -54,20 +81,6 @@ Grive uses cmake to build. Basic install sequence is make -j4 sudo make install -For the detailed instructions, see http://yourcmc.ru/wiki/Grive2#Installation - -When Grive is run for the first time, you should use the "-a" argument to grant -permission to Grive to access to your Google Drive. A URL should be printed. -Go to the link. You will need to login to your Google account if you haven't -done so. After granting the permission to Grive, the browser will show you -an authenication code. Copy-and-paste that to the standard input of Grive. - -If everything works fine, Grive will create .grive and .grive_state files in your -current directory. It will also start downloading files from your Google Drive to -your current directory. - -Enjoy! - ## Version History ### Grive2 v0.5.1-dev From 199a0500996bb3bb8b0349a28e996f269ee747e4 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Thu, 1 Sep 2016 16:19:14 +0300 Subject: [PATCH 153/166] Change deprecated auto_ptr to unique_ptr --- grive/src/main.cc | 11 ++++++----- libgrive/src/base/Drive.cc | 4 ++-- libgrive/src/base/Syncer.hh | 6 +++--- libgrive/src/bfd/SymbolInfo.hh | 2 +- libgrive/src/drive/Syncer1.cc | 12 ++++++------ libgrive/src/drive/Syncer1.hh | 6 +++--- libgrive/src/drive2/Syncer2.cc | 12 ++++++------ libgrive/src/drive2/Syncer2.hh | 6 +++--- libgrive/src/http/CurlAgent.hh | 4 ++-- libgrive/src/http/Download.hh | 2 +- libgrive/src/http/XmlResponse.cc | 2 +- libgrive/src/http/XmlResponse.hh | 9 ++------- libgrive/src/json/JsonParser.hh | 2 +- libgrive/src/json/JsonWriter.hh | 2 +- libgrive/src/json/Val.hh | 2 +- libgrive/src/util/Crypt.hh | 2 +- libgrive/src/util/Function.hh | 2 +- libgrive/src/util/log/CompositeLog.cc | 2 +- libgrive/src/util/log/CompositeLog.hh | 2 +- libgrive/src/util/log/Log.cc | 8 ++++---- libgrive/src/util/log/Log.hh | 2 +- libgrive/src/xml/TreeBuilder.hh | 2 +- libgrive/test/UnitTest.cc | 2 +- 23 files changed, 50 insertions(+), 54 deletions(-) diff --git a/grive/src/main.cc b/grive/src/main.cc index 0894545b..eef90d37 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -66,12 +66,13 @@ void InitGCrypt() void InitLog( const po::variables_map& vm ) { - std::auto_ptr comp_log(new log::CompositeLog) ; - LogBase* console_log = comp_log->Add( std::auto_ptr( new log::DefaultLog ) ) ; + std::unique_ptr comp_log( new log::CompositeLog ) ; + std::unique_ptr def_log( new log::DefaultLog ); + LogBase* console_log = comp_log->Add( def_log ) ; if ( vm.count( "log" ) ) { - std::auto_ptr file_log(new log::DefaultLog( vm["log"].as() )) ; + std::unique_ptr file_log( new log::DefaultLog( vm["log"].as() ) ) ; file_log->Enable( log::debug ) ; file_log->Enable( log::verbose ) ; file_log->Enable( log::info ) ; @@ -96,7 +97,7 @@ void InitLog( const po::variables_map& vm ) console_log->Enable( log::verbose ) ; console_log->Enable( log::debug ) ; } - LogBase::Inst( std::auto_ptr(comp_log.release()) ) ; + LogBase::Inst( comp_log.release() ) ; } int Main( int argc, char **argv ) @@ -151,7 +152,7 @@ int Main( int argc, char **argv ) Log( "config file name %1%", config.Filename(), log::verbose ); - std::auto_ptr http( new http::CurlAgent ); + std::unique_ptr http( new http::CurlAgent ); if ( vm.count( "log-http" ) ) http->SetLog( new http::ResponseLog( vm["log-http"].as(), ".txt" ) ); diff --git a/libgrive/src/base/Drive.cc b/libgrive/src/base/Drive.cc index 5e3eaaf3..fd26225c 100644 --- a/libgrive/src/base/Drive.cc +++ b/libgrive/src/base/Drive.cc @@ -81,7 +81,7 @@ void Drive::DetectChanges() m_state.FromLocal( m_root ) ; Log( "Reading remote server file list", log::info ) ; - std::auto_ptr feed = m_syncer->GetAll() ; + std::unique_ptr feed = m_syncer->GetAll() ; while ( feed->GetNext( m_syncer->Agent() ) ) { @@ -101,7 +101,7 @@ void Drive::ReadChanges() { Trace( "previous change stamp is %1%", prev_stamp ) ; Log( "Detecting changes from last sync", log::info ) ; - std::auto_ptr feed = m_syncer->GetChanges( prev_stamp+1 ) ; + std::unique_ptr feed = m_syncer->GetChanges( prev_stamp+1 ) ; while ( feed->GetNext( m_syncer->Agent() ) ) { std::for_each( diff --git a/libgrive/src/base/Syncer.hh b/libgrive/src/base/Syncer.hh index 29de4b99..237b8949 100644 --- a/libgrive/src/base/Syncer.hh +++ b/libgrive/src/base/Syncer.hh @@ -55,9 +55,9 @@ public : virtual bool Create( Resource *res ) = 0; virtual bool Move( Resource* res, Resource* newParent, std::string newFilename ) = 0; - virtual std::auto_ptr GetFolders() = 0; - virtual std::auto_ptr GetAll() = 0; - virtual std::auto_ptr GetChanges( long min_cstamp ) = 0; + virtual std::unique_ptr GetFolders() = 0; + virtual std::unique_ptr GetAll() = 0; + virtual std::unique_ptr GetChanges( long min_cstamp ) = 0; virtual long GetChangeStamp( long min_cstamp ) = 0; protected: diff --git a/libgrive/src/bfd/SymbolInfo.hh b/libgrive/src/bfd/SymbolInfo.hh index bcbc3b14..9284612d 100644 --- a/libgrive/src/bfd/SymbolInfo.hh +++ b/libgrive/src/bfd/SymbolInfo.hh @@ -54,7 +54,7 @@ public : private : struct Impl ; - const std::auto_ptr m_impl ; + const std::unique_ptr m_impl ; struct BacktraceInfo ; friend struct BacktraceInfo ; diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc index 8eaa9bdd..94997e77 100644 --- a/libgrive/src/drive/Syncer1.cc +++ b/libgrive/src/drive/Syncer1.cc @@ -239,14 +239,14 @@ bool Syncer1::Upload( Resource *res, return true ; } -std::auto_ptr Syncer1::GetFolders() +std::unique_ptr Syncer1::GetFolders() { - return std::auto_ptr( new Feed1( feed_base + "/-/folder?max-results=50&showroot=true" ) ); + return std::unique_ptr( new Feed1( feed_base + "/-/folder?max-results=50&showroot=true" ) ); } -std::auto_ptr Syncer1::GetAll() +std::unique_ptr Syncer1::GetAll() { - return std::auto_ptr( new Feed1( feed_base + "?showfolders=true&showroot=true" ) ); + return std::unique_ptr( new Feed1( feed_base + "?showfolders=true&showroot=true" ) ); } std::string ChangesFeed( int changestamp ) @@ -255,9 +255,9 @@ std::string ChangesFeed( int changestamp ) return changestamp > 0 ? ( feed % changestamp ).str() : feed_changes ; } -std::auto_ptr Syncer1::GetChanges( long min_cstamp ) +std::unique_ptr Syncer1::GetChanges( long min_cstamp ) { - return std::auto_ptr( new Feed1( ChangesFeed( min_cstamp ) ) ); + return std::unique_ptr( new Feed1( ChangesFeed( min_cstamp ) ) ); } long Syncer1::GetChangeStamp( long min_cstamp ) diff --git a/libgrive/src/drive/Syncer1.hh b/libgrive/src/drive/Syncer1.hh index d11c48b5..8c6580db 100644 --- a/libgrive/src/drive/Syncer1.hh +++ b/libgrive/src/drive/Syncer1.hh @@ -38,9 +38,9 @@ public : bool EditContent( Resource *res, bool new_rev ); bool Create( Resource *res ); - std::auto_ptr GetFolders(); - std::auto_ptr GetAll(); - std::auto_ptr GetChanges( long min_cstamp ); + std::unique_ptr GetFolders(); + std::unique_ptr GetAll(); + std::unique_ptr GetChanges( long min_cstamp ); long GetChangeStamp( long min_cstamp ); private : diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index fda18e4b..f7c21d2e 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -211,14 +211,14 @@ bool Syncer2::Upload( Resource *res, bool new_rev ) return true ; } -std::auto_ptr Syncer2::GetFolders() +std::unique_ptr Syncer2::GetFolders() { - return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=trashed%3dfalse+and+mimeType%3d%27" + mime_types::folder + "%27" ) ); + return std::unique_ptr( new Feed2( feeds::files + "?maxResults=1000&q=trashed%3dfalse+and+mimeType%3d%27" + mime_types::folder + "%27" ) ); } -std::auto_ptr Syncer2::GetAll() +std::unique_ptr Syncer2::GetAll() { - return std::auto_ptr( new Feed2( feeds::files + "?maxResults=1000&q=trashed%3dfalse" ) ); + return std::unique_ptr( new Feed2( feeds::files + "?maxResults=1000&q=trashed%3dfalse" ) ); } std::string ChangesFeed( long changestamp, int maxResults = 1000 ) @@ -227,9 +227,9 @@ std::string ChangesFeed( long changestamp, int maxResults = 1000 ) return ( changestamp > 0 ? feed % maxResults % changestamp : feed % maxResults ).str() ; } -std::auto_ptr Syncer2::GetChanges( long min_cstamp ) +std::unique_ptr Syncer2::GetChanges( long min_cstamp ) { - return std::auto_ptr( new Feed2( ChangesFeed( min_cstamp ) ) ); + return std::unique_ptr( new Feed2( ChangesFeed( min_cstamp ) ) ); } long Syncer2::GetChangeStamp( long min_cstamp ) diff --git a/libgrive/src/drive2/Syncer2.hh b/libgrive/src/drive2/Syncer2.hh index 8a34757e..e62d8b55 100644 --- a/libgrive/src/drive2/Syncer2.hh +++ b/libgrive/src/drive2/Syncer2.hh @@ -39,9 +39,9 @@ public : bool Create( Resource *res ); bool Move( Resource* res, Resource* newParent, std::string newFilename ); - std::auto_ptr GetFolders(); - std::auto_ptr GetAll(); - std::auto_ptr GetChanges( long min_cstamp ); + std::unique_ptr GetFolders(); + std::unique_ptr GetAll(); + std::unique_ptr GetChanges( long min_cstamp ); long GetChangeStamp( long min_cstamp ); private : diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index ac7fe5b1..c25fa1a0 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -72,8 +72,8 @@ private : private : struct Impl ; - std::auto_ptr m_pimpl ; - std::auto_ptr m_log ; + std::unique_ptr m_pimpl ; + std::unique_ptr m_log ; } ; } } // end of namespace diff --git a/libgrive/src/http/Download.hh b/libgrive/src/http/Download.hh index 77cc1ae8..fdb75800 100644 --- a/libgrive/src/http/Download.hh +++ b/libgrive/src/http/Download.hh @@ -48,7 +48,7 @@ public : private : File m_file ; - std::auto_ptr m_crypt ; + std::unique_ptr m_crypt ; } ; } } // end of namespace diff --git a/libgrive/src/http/XmlResponse.cc b/libgrive/src/http/XmlResponse.cc index 3df42f94..3c9d6f33 100644 --- a/libgrive/src/http/XmlResponse.cc +++ b/libgrive/src/http/XmlResponse.cc @@ -30,7 +30,7 @@ XmlResponse::XmlResponse() : m_tb( new xml::TreeBuilder ) void XmlResponse::Clear() { - m_tb.reset(new xml::TreeBuilder); + m_tb.reset(new xml::TreeBuilder); } std::size_t XmlResponse::Write( const char *data, std::size_t count ) diff --git a/libgrive/src/http/XmlResponse.hh b/libgrive/src/http/XmlResponse.hh index b1505097..26736e46 100644 --- a/libgrive/src/http/XmlResponse.hh +++ b/libgrive/src/http/XmlResponse.hh @@ -20,15 +20,10 @@ #pragma once #include "util/DataStream.hh" +#include "xml/TreeBuilder.hh" #include -namespace gr { namespace xml -{ - class Node ; - class TreeBuilder ; -} } - namespace gr { namespace http { class XmlResponse : public DataStream @@ -44,7 +39,7 @@ public : xml::Node Response() const ; private : - std::auto_ptr m_tb ; + std::unique_ptr m_tb ; } ; } } // end of namespace diff --git a/libgrive/src/json/JsonParser.hh b/libgrive/src/json/JsonParser.hh index ea9e09ed..5842699d 100644 --- a/libgrive/src/json/JsonParser.hh +++ b/libgrive/src/json/JsonParser.hh @@ -50,7 +50,7 @@ public : private : struct Impl ; - std::auto_ptr m_impl ; + std::unique_ptr m_impl ; } ; } // end of namespace diff --git a/libgrive/src/json/JsonWriter.hh b/libgrive/src/json/JsonWriter.hh index 66f95bf9..595f8e06 100644 --- a/libgrive/src/json/JsonWriter.hh +++ b/libgrive/src/json/JsonWriter.hh @@ -51,7 +51,7 @@ private : private : struct Impl ; - std::auto_ptr m_impl ; + std::unique_ptr m_impl ; } ; std::string WriteJson( const Val& val ); diff --git a/libgrive/src/json/Val.hh b/libgrive/src/json/Val.hh index 4c8b9ccd..eace6a29 100644 --- a/libgrive/src/json/Val.hh +++ b/libgrive/src/json/Val.hh @@ -130,7 +130,7 @@ private : template struct Impl ; - std::auto_ptr m_base ; + std::unique_ptr m_base ; private : void Select( const Object& obj, const std::string& key, std::vector& result ) const ; diff --git a/libgrive/src/util/Crypt.hh b/libgrive/src/util/Crypt.hh index dcd8dd34..a9558096 100644 --- a/libgrive/src/util/Crypt.hh +++ b/libgrive/src/util/Crypt.hh @@ -50,7 +50,7 @@ public : private : struct Impl ; - std::auto_ptr m_impl ; + std::unique_ptr m_impl ; } ; } } // end of namespace gr diff --git a/libgrive/src/util/Function.hh b/libgrive/src/util/Function.hh index 197418f0..d5d91147 100644 --- a/libgrive/src/util/Function.hh +++ b/libgrive/src/util/Function.hh @@ -178,7 +178,7 @@ public : private : typedef impl::FuncImpl Impl ; - std::auto_ptr m_pimpl ; + std::unique_ptr m_pimpl ; } ; } // end of namespace diff --git a/libgrive/src/util/log/CompositeLog.cc b/libgrive/src/util/log/CompositeLog.cc index 6c9140ab..9de64681 100644 --- a/libgrive/src/util/log/CompositeLog.cc +++ b/libgrive/src/util/log/CompositeLog.cc @@ -39,7 +39,7 @@ CompositeLog::~CompositeLog() std::for_each( m_logs.begin(), m_logs.end(), Destroy() ) ; } -LogBase* CompositeLog::Add( std::auto_ptr log ) +LogBase* CompositeLog::Add( std::unique_ptr& log ) { m_logs.push_back( log.get() ) ; return log.release() ; diff --git a/libgrive/src/util/log/CompositeLog.hh b/libgrive/src/util/log/CompositeLog.hh index 6efe3be0..80b41e41 100644 --- a/libgrive/src/util/log/CompositeLog.hh +++ b/libgrive/src/util/log/CompositeLog.hh @@ -32,7 +32,7 @@ public : CompositeLog() ; ~CompositeLog() ; - LogBase* Add( std::auto_ptr log ) ; + LogBase* Add( std::unique_ptr& log ) ; void Log( const log::Fmt& msg, log::Serverity s ) ; diff --git a/libgrive/src/util/log/Log.cc b/libgrive/src/util/log/Log.cc index b93e1b7f..07bc0860 100644 --- a/libgrive/src/util/log/Log.cc +++ b/libgrive/src/util/log/Log.cc @@ -40,12 +40,12 @@ public : } } ; -LogBase* LogBase::Inst( std::auto_ptr log ) +LogBase* LogBase::Inst( LogBase *log ) { - static std::auto_ptr inst( new MockLog ) ; + static std::unique_ptr inst( new MockLog ) ; - if ( log.get() != 0 ) - inst = log ; + if ( log != 0 ) + inst.reset( log ) ; assert( inst.get() != 0 ) ; return inst.get() ; diff --git a/libgrive/src/util/log/Log.hh b/libgrive/src/util/log/Log.hh index 762320c3..25760e51 100644 --- a/libgrive/src/util/log/Log.hh +++ b/libgrive/src/util/log/Log.hh @@ -65,7 +65,7 @@ public : virtual bool Enable( log::Serverity s, bool enable = true ) = 0 ; virtual bool IsEnabled( log::Serverity s ) const = 0 ; - static LogBase* Inst( std::auto_ptr log = std::auto_ptr() ) ; + static LogBase* Inst( LogBase *log = 0 ) ; virtual ~LogBase() ; protected : diff --git a/libgrive/src/xml/TreeBuilder.hh b/libgrive/src/xml/TreeBuilder.hh index fe71a996..2e958e7e 100644 --- a/libgrive/src/xml/TreeBuilder.hh +++ b/libgrive/src/xml/TreeBuilder.hh @@ -55,7 +55,7 @@ private : private : struct Impl ; - std::auto_ptr m_impl ; + std::unique_ptr m_impl ; } ; } } // end of namespace diff --git a/libgrive/test/UnitTest.cc b/libgrive/test/UnitTest.cc index c229ae60..7dfbf772 100644 --- a/libgrive/test/UnitTest.cc +++ b/libgrive/test/UnitTest.cc @@ -35,7 +35,7 @@ int main( int argc, char **argv ) { using namespace grut ; - gr::LogBase::Inst( std::auto_ptr(new gr::log::DefaultLog) ) ; + gr::LogBase::Inst( new gr::log::DefaultLog ) ; CppUnit::TextUi::TestRunner runner; runner.addTest( Entry1Test::suite( ) ) ; From d35c84946852aadbc1e5ebc8c338357a69f8de00 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 2 Sep 2016 13:47:08 +0300 Subject: [PATCH 154/166] add std=c++0x for older gcc (fixes #96) --- CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 7b330d65..95443aa7 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -5,7 +5,7 @@ set( GRIVE_VERSION "0.5.1-dev" ) # common compile options add_definitions( -DVERSION="${GRIVE_VERSION}" ) -add_definitions( -D_FILE_OFFSET_BITS=64 ) +add_definitions( -D_FILE_OFFSET_BITS=64 -std=c++0x ) add_subdirectory( libgrive ) add_subdirectory( grive ) From 59d02a65cbafce3b64c74728bc18c2c1444939c6 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 28 Sep 2016 01:01:50 +0300 Subject: [PATCH 155/166] Remove outdated Doclist API implementation --- libgrive/CMakeLists.txt | 8 - libgrive/src/drive/CommonUri.hh | 34 ---- libgrive/src/drive/Entry1.cc | 86 ---------- libgrive/src/drive/Entry1.hh | 42 ----- libgrive/src/drive/Feed1.cc | 64 -------- libgrive/src/drive/Feed1.hh | 40 ----- libgrive/src/drive/Syncer1.cc | 271 ------------------------------- libgrive/src/drive/Syncer1.hh | 52 ------ libgrive/test/drive/EntryTest.cc | 59 ------- libgrive/test/drive/EntryTest.hh | 41 ----- 10 files changed, 697 deletions(-) delete mode 100644 libgrive/src/drive/CommonUri.hh delete mode 100644 libgrive/src/drive/Entry1.cc delete mode 100644 libgrive/src/drive/Entry1.hh delete mode 100644 libgrive/src/drive/Feed1.cc delete mode 100644 libgrive/src/drive/Feed1.hh delete mode 100644 libgrive/src/drive/Syncer1.cc delete mode 100644 libgrive/src/drive/Syncer1.hh delete mode 100644 libgrive/test/drive/EntryTest.cc delete mode 100644 libgrive/test/drive/EntryTest.hh diff --git a/libgrive/CMakeLists.txt b/libgrive/CMakeLists.txt index 39d64f72..54776bd1 100644 --- a/libgrive/CMakeLists.txt +++ b/libgrive/CMakeLists.txt @@ -49,10 +49,6 @@ include_directories( ${YAJL_INCLUDE_DIRS} ) -file(GLOB DRIVE_HEADERS - ${libgrive_SOURCE_DIR}/src/drive/*.hh -) - file (GLOB PROTOCOL_HEADERS ${libgrive_SOURCE_DIR}/src/protocol/*.hh ) @@ -67,14 +63,12 @@ file (GLOB XML_HEADERS file (GLOB LIBGRIVE_SRC src/base/*.cc - src/drive/*.cc src/drive2/*.cc src/http/*.cc src/protocol/*.cc src/json/*.cc src/util/*.cc src/util/log/*.cc - src/xml/*.cc ) add_definitions( @@ -121,9 +115,7 @@ IF ( CPPUNIT_FOUND ) # list of test source files here file(GLOB TEST_SRC test/base/*.cc - test/drive/*.cc test/util/*.cc - test/xml/*.cc ) add_executable( unittest diff --git a/libgrive/src/drive/CommonUri.hh b/libgrive/src/drive/CommonUri.hh deleted file mode 100644 index bbc3b0b6..00000000 --- a/libgrive/src/drive/CommonUri.hh +++ /dev/null @@ -1,34 +0,0 @@ -/* - Common URIs for the old "Document List" Google Docs API - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include - -namespace gr { namespace v1 -{ - const std::string feed_base = "https://docs.google.com/feeds/default/private/full" ; - const std::string feed_changes = "https://docs.google.com/feeds/default/private/changes" ; - const std::string feed_metadata = "https://docs.google.com/feeds/metadata/default" ; - - const std::string root_href = - "https://docs.google.com/feeds/default/private/full/folder%3Aroot" ; - const std::string root_create = - "https://docs.google.com/feeds/upload/create-session/default/private/full" ; -} } diff --git a/libgrive/src/drive/Entry1.cc b/libgrive/src/drive/Entry1.cc deleted file mode 100644 index 9f75138a..00000000 --- a/libgrive/src/drive/Entry1.cc +++ /dev/null @@ -1,86 +0,0 @@ -/* - Item class implementation for the old "Document List" Google Docs API - Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#include "Entry1.hh" -#include "CommonUri.hh" - -#include "util/Crypt.hh" -#include "util/log/Log.hh" -#include "util/OS.hh" -#include "xml/Node.hh" -#include "xml/NodeSet.hh" - -#include -#include - -namespace gr { namespace v1 { - -Entry1::Entry1(): - Entry () -{ - m_self_href = root_href; -} - -/// construct an entry for remote - Doclist API v3 -Entry1::Entry1( const xml::Node& n ) -{ - Update( n ) ; -} - -void Entry1::Update( const xml::Node& n ) -{ - m_title = n["title"] ; - m_etag = n["@gd:etag"] ; - m_filename = n["docs:suggestedFilename"] ; - m_content_src = n["content"]["@src"] ; - m_self_href = n["link"].Find( "@rel", "self" )["@href"] ; - m_mtime = DateTime( n["updated"] ) ; - - m_resource_id = n["gd:resourceId"] ; - m_md5 = n["docs:md5Checksum"] ; - m_is_dir = n["category"].Find( "@scheme", "http://schemas.google.com/g/2005#kind" )["@label"] == "folder" ; - m_is_editable = !n["link"].Find( "@rel", m_is_dir - ? "http://schemas.google.com/g/2005#resumable-create-media" : "http://schemas.google.com/g/2005#resumable-edit-media" ) - ["@href"].empty() ; - - // changestamp only appear in change feed entries - xml::NodeSet cs = n["docs:changestamp"]["@value"] ; - m_change_stamp = cs.empty() ? -1 : std::atoi( cs.front().Value().c_str() ) ; - if ( m_change_stamp != -1 ) - { - m_self_href = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#alt-self" )["@href"] ; - } - - m_parent_hrefs.clear( ) ; - xml::NodeSet parents = n["link"].Find( "@rel", "http://schemas.google.com/docs/2007#parent" ) ; - for ( xml::NodeSet::iterator i = parents.begin() ; i != parents.end() ; ++i ) - { - std::string href = (*i)["@href"]; - if ( href == root_href ) - href = "root"; // API-independent root href - m_parent_hrefs.push_back( href ) ; - } - - // convert to lower case for easy comparison - std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; - - m_is_removed = !n["gd:deleted"].empty() || !n["docs:removed"].empty() ; -} - -} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Entry1.hh b/libgrive/src/drive/Entry1.hh deleted file mode 100644 index 6fff374e..00000000 --- a/libgrive/src/drive/Entry1.hh +++ /dev/null @@ -1,42 +0,0 @@ -/* - Item class implementation for the old "Document List" Google Docs API - Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include "base/Entry.hh" - -namespace gr { - -namespace xml -{ - class Node ; -} - -namespace v1 { - -class Entry1: public Entry -{ -public : - Entry1( ) ; - explicit Entry1( const xml::Node& n ) ; -private : - void Update( const xml::Node& entry ) ; -} ; - -} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed1.cc b/libgrive/src/drive/Feed1.cc deleted file mode 100644 index 86d59bd7..00000000 --- a/libgrive/src/drive/Feed1.cc +++ /dev/null @@ -1,64 +0,0 @@ -/* - Item list ("Feed") implementation for the old "Document List" Google Docs API - Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#include "CommonUri.hh" -#include "Feed1.hh" - -#include "Entry1.hh" - -#include "http/Agent.hh" -#include "http/Header.hh" -#include "http/XmlResponse.hh" -#include "xml/NodeSet.hh" - -#include - -#include - -namespace gr { namespace v1 { - -Feed1::Feed1( const std::string &url ): - Feed( url ) -{ -} - -bool Feed1::GetNext( http::Agent *http ) -{ - http::XmlResponse xrsp ; - - if ( m_next.empty() ) - return false; - - http->Get( m_next, &xrsp, http::Header() ) ; - - xml::Node m_root = xrsp.Response() ; - xml::NodeSet xe = m_root["entry"] ; - m_entries.clear() ; - for ( xml::NodeSet::iterator i = xe.begin() ; i != xe.end() ; ++i ) - { - m_entries.push_back( Entry1( *i ) ); - } - - xml::NodeSet nss = m_root["link"].Find( "@rel", "next" ) ; - m_next = nss.empty() ? std::string( "" ) : nss["@href"]; - - return true; -} - -} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Feed1.hh b/libgrive/src/drive/Feed1.hh deleted file mode 100644 index 40f88a8f..00000000 --- a/libgrive/src/drive/Feed1.hh +++ /dev/null @@ -1,40 +0,0 @@ -/* - Item list ("Feed") implementation for the old "Document List" Google Docs API - Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include "base/Feed.hh" - -#include "xml/Node.hh" -#include "xml/NodeSet.hh" - -#include - -#include - -namespace gr { namespace v1 { - -class Feed1: public Feed -{ -public : - Feed1( const std::string& url ) ; - bool GetNext( http::Agent *http ) ; -} ; - -} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Syncer1.cc b/libgrive/src/drive/Syncer1.cc deleted file mode 100644 index 94997e77..00000000 --- a/libgrive/src/drive/Syncer1.cc +++ /dev/null @@ -1,271 +0,0 @@ -/* - Syncer implementation for the old "Document List" Google Docs API - Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#include "base/Resource.hh" -#include "CommonUri.hh" -#include "Entry1.hh" -#include "Feed1.hh" -#include "Syncer1.hh" - -#include "http/Agent.hh" -#include "http/Header.hh" -#include "http/StringResponse.hh" -#include "http/XmlResponse.hh" - -#include "xml/Node.hh" -#include "xml/NodeSet.hh" -#include "xml/String.hh" -#include "xml/TreeBuilder.hh" - -#include "util/File.hh" -#include "util/OS.hh" -#include "util/log/Log.hh" - -#include - -#include - -// for debugging -#include - -namespace gr { namespace v1 { - -// hard coded XML file -const std::string xml_meta = - "\n" - "" - "" - "%2%" - "" ; - -Syncer1::Syncer1( http::Agent *http ): - Syncer( http ) -{ - assert( http != 0 ) ; -} - -void Syncer1::DeleteRemote( Resource *res ) -{ - http::StringResponse str ; - - try - { - http::Header hdr ; - hdr.Add( "If-Match: " + res->ETag() ) ; - - // don't know why, but an update before deleting seems to work always - http::XmlResponse xml ; - m_http->Get( res->SelfHref(), &xml, hdr ) ; - AssignIDs( res, Entry1( xml.Response() ) ) ; - - m_http->Request( "DELETE", res->SelfHref(), NULL, &str, hdr ) ; - } - catch ( Exception& e ) - { - // don't rethrow here. there are some cases that I don't know why - // the delete will fail. - Trace( "Exception %1% %2%", - boost::diagnostic_information(e), - str.Response() ) ; - } -} - -bool Syncer1::EditContent( Resource *res, bool new_rev ) -{ - assert( res->Parent() ) ; - assert( res->Parent()->GetState() == Resource::sync ) ; - - if ( !res->IsEditable() ) - { - Log( "Cannot upload %1%: file read-only. %2%", res->Name(), res->StateStr(), log::warning ) ; - return false ; - } - - return Upload( res, feed_base + "/" + res->ResourceID() + ( new_rev ? "?new-revision=true" : "" ), false ) ; -} - -bool Syncer1::Create( Resource *res ) -{ - assert( res->Parent() ) ; - assert( res->Parent()->IsFolder() ) ; - assert( res->Parent()->GetState() == Resource::sync ) ; - - if ( res->IsFolder() ) - { - std::string uri = feed_base ; - if ( !res->Parent()->IsRoot() ) - uri += ( "/" + m_http->Escape( res->Parent()->ResourceID() ) + "/contents" ) ; - - std::string meta = (boost::format( xml_meta ) - % "folder" - % xml::Escape( res->Name() ) - ).str() ; - - http::Header hdr ; - hdr.Add( "Content-Type: application/atom+xml" ) ; - - http::XmlResponse xml ; - m_http->Post( uri, meta, &xml, hdr ) ; - AssignIDs( res, Entry1( xml.Response() ) ) ; - - return true ; - } - else if ( res->Parent()->IsEditable() ) - { - return Upload( res, root_create + (res->Parent()->ResourceID() == "folder:root" - ? "" : "/" + res->Parent()->ResourceID() + "/contents") + "?convert=false", true ) ; - } - else - { - Log( "parent of %1% does not exist: cannot upload", res->Name(), log::warning ) ; - return false ; - } -} - -bool Syncer1::Upload( Resource *res, - const std::string& link, - bool post ) -{ - File file( res->Path() ) ; - std::ostringstream xcontent_len ; - xcontent_len << "X-Upload-Content-Length: " << file.Size() ; - - http::Header hdr ; - hdr.Add( "Content-Type: application/atom+xml" ) ; - hdr.Add( "X-Upload-Content-Type: application/octet-stream" ) ; - hdr.Add( xcontent_len.str() ) ; - hdr.Add( "If-Match: " + res->ETag() ) ; - hdr.Add( "Expect:" ) ; - - std::string meta = (boost::format( xml_meta ) - % res->Kind() - % xml::Escape( res->Name() ) - ).str() ; - - bool retrying = false; - while ( true ) - { - if ( retrying ) - { - file.Seek( 0, SEEK_SET ); - os::Sleep( 5 ); - } - - try - { - http::StringResponse str ; - if ( post ) - m_http->Post( link, meta, &str, hdr ) ; - else - m_http->Put( link, meta, &str, hdr ) ; - } - catch ( Exception &e ) - { - std::string const *info = boost::get_error_info(e); - if ( info && (*info == "XML_Parse") ) - { - Log( "Error parsing pre-upload response XML, retrying whole upload in 5s", - log::warning ); - retrying = true; - continue; - } - else - { - throw e; - } - } - - http::Header uphdr ; - uphdr.Add( "Expect:" ) ; - uphdr.Add( "Accept:" ) ; - - // the content upload URL is in the "Location" HTTP header - std::string uplink = m_http->RedirLocation() ; - http::XmlResponse xml ; - - long http_code = 0; - try - { - http_code = m_http->Put( uplink, &file, &xml, uphdr ) ; - } - catch ( Exception &e ) - { - std::string const *info = boost::get_error_info(e); - if ( info && (*info == "XML_Parse") ) - { - Log( "Error parsing response XML, retrying whole upload in 5s", - log::warning ); - retrying = true; - continue; - } - else - { - throw e; - } - } - - if ( http_code == 410 || http_code == 412 ) - { - Log( "request failed with %1%, body: %2%, retrying whole upload in 5s", http_code, m_http->LastError(), log::warning ) ; - retrying = true; - continue; - } - - if ( retrying ) - Log( "upload succeeded on retry", log::warning ); - Entry1 responseEntry = Entry1( xml.Response() ); - AssignIDs( res, responseEntry ) ; - res->SetServerTime( responseEntry.MTime() ); - break; - } - - return true ; -} - -std::unique_ptr Syncer1::GetFolders() -{ - return std::unique_ptr( new Feed1( feed_base + "/-/folder?max-results=50&showroot=true" ) ); -} - -std::unique_ptr Syncer1::GetAll() -{ - return std::unique_ptr( new Feed1( feed_base + "?showfolders=true&showroot=true" ) ); -} - -std::string ChangesFeed( int changestamp ) -{ - boost::format feed( feed_changes + "?start-index=%1%" ) ; - return changestamp > 0 ? ( feed % changestamp ).str() : feed_changes ; -} - -std::unique_ptr Syncer1::GetChanges( long min_cstamp ) -{ - return std::unique_ptr( new Feed1( ChangesFeed( min_cstamp ) ) ); -} - -long Syncer1::GetChangeStamp( long min_cstamp ) -{ - http::XmlResponse xrsp ; - m_http->Get( ChangesFeed( min_cstamp ), &xrsp, http::Header() ) ; - - return std::atoi( xrsp.Response()["docs:largestChangestamp"]["@value"].front().Value().c_str() ); -} - -} } // end of namespace gr::v1 diff --git a/libgrive/src/drive/Syncer1.hh b/libgrive/src/drive/Syncer1.hh deleted file mode 100644 index 8c6580db..00000000 --- a/libgrive/src/drive/Syncer1.hh +++ /dev/null @@ -1,52 +0,0 @@ -/* - Syncer implementation for the old "Document List" Google Docs API - Copyright (C) 2012 Wan Wai Ho, (C) 2015 Vitaliy Filippov - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include "base/Syncer.hh" - -namespace gr { - -class Feed; - -namespace v1 { - -class Syncer1: public Syncer -{ - -public : - - Syncer1( http::Agent *http ); - - void DeleteRemote( Resource *res ); - bool EditContent( Resource *res, bool new_rev ); - bool Create( Resource *res ); - - std::unique_ptr GetFolders(); - std::unique_ptr GetAll(); - std::unique_ptr GetChanges( long min_cstamp ); - long GetChangeStamp( long min_cstamp ); - -private : - - bool Upload( Resource *res, const std::string& link, bool post); - -} ; - -} } // end of namespace gr::v1 diff --git a/libgrive/test/drive/EntryTest.cc b/libgrive/test/drive/EntryTest.cc deleted file mode 100644 index a78516db..00000000 --- a/libgrive/test/drive/EntryTest.cc +++ /dev/null @@ -1,59 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#include "EntryTest.hh" - -#include "Assert.hh" - -#include "drive/Entry1.hh" -#include "xml/Node.hh" -#include "xml/NodeSet.hh" -#include "xml/TreeBuilder.hh" - -#include - -namespace grut { - -using namespace gr ; -using namespace gr::v1 ; - -Entry1Test::Entry1Test( ) -{ -} - -void Entry1Test::TestXml( ) -{ - xml::Node root = xml::TreeBuilder::ParseFile( TEST_DATA "entry.xml" ) ; - - CPPUNIT_ASSERT( !root["entry"].empty() ) ; - - Entry1 subject( root["entry"].front() ) ; - GRUT_ASSERT_EQUAL( "snes", subject.Title() ) ; - GRUT_ASSERT_EQUAL( "\"WxYPGE8CDyt7ImBk\"", subject.ETag() ) ; - GRUT_ASSERT_EQUAL( "https://docs.google.com/feeds/default/private/full/folder%3A0B5KhdsbryVeGMl83OEV1ZVc3cUE", - subject.SelfHref() ) ; - - GRUT_ASSERT_EQUAL( 1U, subject.ParentHrefs().size() ) ; - GRUT_ASSERT_EQUAL( "https://docs.google.com/feeds/default/private/full/folder%3A0B5KhdsbryVeGNEZjdUxzZHl3Sjg", - subject.ParentHrefs().front() ) ; - - GRUT_ASSERT_EQUAL( true, subject.IsDir() ) ; -} - -} // end of namespace grut diff --git a/libgrive/test/drive/EntryTest.hh b/libgrive/test/drive/EntryTest.hh deleted file mode 100644 index f7bc8f51..00000000 --- a/libgrive/test/drive/EntryTest.hh +++ /dev/null @@ -1,41 +0,0 @@ -/* - grive: an GPL program to sync a local directory with Google Drive - Copyright (C) 2012 Wan Wai Ho - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License - as published by the Free Software Foundation version 2 - of the License. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -*/ - -#pragma once - -#include -#include - -namespace grut { - -class Entry1Test : public CppUnit::TestFixture -{ -public : - Entry1Test( ) ; - - // declare suit function - CPPUNIT_TEST_SUITE( Entry1Test ) ; - CPPUNIT_TEST( TestXml ) ; - CPPUNIT_TEST_SUITE_END(); - -private : - void TestXml( ) ; -} ; - -} // end of namespace From dd77c998728cd0f72ddc5fc2a4f1cfa6c5f139c3 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 28 Sep 2016 01:19:13 +0300 Subject: [PATCH 156/166] Check MD5 only when it is needed (fixes #101) This makes grive check md5 sums only when: 1) a local rename is supposed (when there are a new file and a deleted file of the same size) 2) local ctime is changed, but file size isn't --- README.md | 2 + libgrive/src/base/Entry.cc | 8 ++- libgrive/src/base/Entry.hh | 3 + libgrive/src/base/Resource.cc | 112 ++++++++++++++++++++---------- libgrive/src/base/Resource.hh | 5 ++ libgrive/src/base/ResourceTree.cc | 8 ++- libgrive/src/base/ResourceTree.hh | 5 ++ libgrive/src/base/State.cc | 26 +++---- libgrive/src/drive2/Entry2.cc | 2 + libgrive/src/util/OS.cc | 8 +-- libgrive/src/util/OS.hh | 4 +- 11 files changed, 128 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index 9cbc4e83..89bfec19 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,8 @@ Grive uses cmake to build. Basic install sequence is - ignore regexp does not persist anymore (note that Grive will still track it to not accidentally delete remote files when changing ignore regexp) - added options to limit upload and download speed +- faster upload of new and changed files. now Grive uploads files without first calculating + md5 checksum when file is created locally or when its size changes. ### Grive2 v0.5 diff --git a/libgrive/src/base/Entry.cc b/libgrive/src/base/Entry.cc index ed25a101..05f346bc 100644 --- a/libgrive/src/base/Entry.cc +++ b/libgrive/src/base/Entry.cc @@ -36,7 +36,8 @@ Entry::Entry( ) : m_is_dir ( true ), m_resource_id ( "folder:root" ), m_change_stamp ( -1 ), - m_is_removed ( false ) + m_is_removed ( false ), + m_size ( 0 ) { } @@ -65,6 +66,11 @@ std::string Entry::MD5() const return m_md5 ; } +u64_t Entry::Size() const +{ + return m_size ; +} + DateTime Entry::MTime() const { return m_mtime ; diff --git a/libgrive/src/base/Entry.hh b/libgrive/src/base/Entry.hh index 6472e892..8324f6db 100644 --- a/libgrive/src/base/Entry.hh +++ b/libgrive/src/base/Entry.hh @@ -19,6 +19,7 @@ #pragma once +#include "util/Types.hh" #include "util/DateTime.hh" #include "util/FileSystem.hh" @@ -44,6 +45,7 @@ public : bool IsDir() const ; std::string MD5() const ; DateTime MTime() const ; + u64_t Size() const ; std::string Name() const ; @@ -80,6 +82,7 @@ protected : DateTime m_mtime ; bool m_is_removed ; + u64_t m_size ; } ; } // end of namespace gr diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index 72c3a267..b03aa5c9 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -47,6 +47,7 @@ namespace gr { Resource::Resource( const fs::path& root_folder ) : m_name ( root_folder.string() ), m_kind ( "folder" ), + m_size ( 0 ), m_id ( "folder:root" ), m_href ( "root" ), m_is_editable( true ), @@ -60,6 +61,7 @@ Resource::Resource( const fs::path& root_folder ) : Resource::Resource( const std::string& name, const std::string& kind ) : m_name ( name ), m_kind ( kind ), + m_size ( 0 ), m_is_editable( true ), m_parent ( 0 ), m_state ( unknown ), @@ -149,6 +151,7 @@ void Resource::AssignIDs( const Entry& remote ) m_content = remote.ContentSrc() ; m_is_editable = remote.IsEditable() ; m_etag = remote.ETag() ; + m_md5 = remote.MD5() ; } } @@ -193,7 +196,7 @@ void Resource::FromRemoteFile( const Entry& remote ) m_state = local_deleted ; } } - + // remote checksum unknown, assume the file is not changed in remote else if ( remote.MD5().empty() ) { @@ -201,16 +204,9 @@ void Resource::FromRemoteFile( const Entry& remote ) Path(), log::verbose ) ; m_state = sync ; } - - // if checksum is equal, no need to compare the mtime - else if ( remote.MD5() == m_md5 ) - { - Log( "file %1% is already in sync", Path(), log::verbose ) ; - m_state = sync ; - } // use mtime to check which one is more recent - else + else if ( remote.Size() != m_size || remote.MD5() != GetMD5() ) { assert( m_state != unknown ) ; @@ -230,6 +226,13 @@ void Resource::FromRemoteFile( const Entry& remote ) else Trace( "file %1% state is %2%", m_name, m_state ) ; } + + // if checksum is equal, no need to compare the mtime + else + { + Log( "file %1% is already in sync", Path(), log::verbose ) ; + m_state = sync ; + } } void Resource::FromDeleted( Val& state ) @@ -242,6 +245,8 @@ void Resource::FromDeleted( Val& state ) m_md5 = state["md5"]; if ( state.Has( "srv_time" ) ) m_mtime.Assign( state[ "srv_time" ].U64(), 0 ) ; + if ( state.Has( "size" ) ) + m_size = state[ "size" ].U64(); m_state = both_deleted; } @@ -259,7 +264,7 @@ void Resource::FromLocal( Val& state ) bool is_dir; try { - os::Stat( path, &m_ctime, NULL, &is_dir ) ; + os::Stat( path, &m_ctime, (off64_t*)&m_size, &is_dir ) ; } catch ( os::Error &e ) { @@ -287,9 +292,9 @@ void Resource::FromLocal( Val& state ) { if ( !is_dir ) { - m_md5 = crypt::MD5::Get( path ); // File is changed locally. TODO: Detect conflicts - is_changed = !state.Has( "md5" ) || m_md5 != state["md5"].Str(); + is_changed = ( state.Has( "size" ) && m_size != state["size"].U64() ) || + !state.Has( "md5" ) || GetMD5() != state["md5"].Str(); } else is_changed = true; @@ -479,30 +484,33 @@ void Resource::Sync( Syncer *syncer, ResourceTree *res_tree, const Val& options } } -void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& options ) +bool Resource::CheckRename( Syncer* syncer, ResourceTree *res_tree ) { - assert( !IsRoot() || m_state == sync ) ; // root is always sync - assert( IsRoot() || !syncer || m_parent->IsFolder() ) ; - assert( IsRoot() || m_parent->m_state != remote_deleted ) ; - assert( IsRoot() || m_parent->m_state != local_deleted ) ; - - const fs::path path = Path() ; - - // Detect renames - if ( !IsFolder() && ( m_state == local_new || m_state == local_deleted || - m_state == remote_new || m_state == remote_deleted ) ) + if ( !IsFolder() && ( m_state == local_new || m_state == remote_new ) ) { - details::MD5Range moved = res_tree->FindByMD5( m_md5 ); - bool is_local = m_state == local_new || m_state == local_deleted; - State other; - if ( m_state == local_new ) - other = local_deleted; - else if ( m_state == local_deleted ) - other = local_new; - else if ( m_state == remote_new ) - other = remote_deleted; - else - other = remote_new; + bool is_local = m_state == local_new; + State other = is_local ? local_deleted : remote_deleted; + if ( is_local ) + { + // First check size index for locally added files + details::SizeRange moved = res_tree->FindBySize( m_size ); + bool found = false; + for ( details::SizeMap::iterator i = moved.first ; i != moved.second; i++ ) + { + Resource *m = *i; + if ( m->m_state == other ) + { + found = true; + break; + } + } + if ( !found ) + { + // Don't check md5 sums if there are no deleted files with same size + return false; + } + } + details::MD5Range moved = res_tree->FindByMD5( GetMD5() ); for ( details::MD5Map::iterator i = moved.first ; i != moved.second; i++ ) { Resource *m = *i; @@ -530,10 +538,25 @@ void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& opti } from->m_state = both_deleted; to->m_state = sync; - return; + return true; } } } + return false; +} + +void Resource::SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& options ) +{ + assert( !IsRoot() || m_state == sync ) ; // root is always sync + assert( IsRoot() || !syncer || m_parent->IsFolder() ) ; + assert( IsRoot() || m_parent->m_state != remote_deleted ) ; + assert( IsRoot() || m_parent->m_state != local_deleted ) ; + + const fs::path path = Path() ; + + // Detect renames + if ( CheckRename( syncer, res_tree ) ) + return; switch ( m_state ) { @@ -688,6 +711,7 @@ void Resource::SetIndex( bool re_stat ) if ( !is_dir ) { m_json->Set( "md5", Val( m_md5 ) ); + m_json->Set( "size", Val( m_size ) ); m_json->Del( "tree" ); } else @@ -695,6 +719,7 @@ void Resource::SetIndex( bool re_stat ) // add tree item if it does not exist m_json->Item( "tree" ); m_json->Del( "md5" ); + m_json->Del( "size" ); } } @@ -731,11 +756,28 @@ std::string Resource::StateStr() const return ss.str() ; } +u64_t Resource::Size() const +{ + return m_size ; +} + std::string Resource::MD5() const { return m_md5 ; } +std::string Resource::GetMD5() +{ + if ( m_md5.empty() && !IsFolder() && m_local_exists ) + { + // MD5 checksum is calculated lazily and only when really needed: + // 1) when a local rename is supposed (when there are a new file and a deleted file of the same size) + // 2) when local ctime is changed, but file size isn't + m_md5 = crypt::MD5::Get( Path() ); + } + return m_md5 ; +} + bool Resource::IsRoot() const { // Root entry does not show up in file feeds, so we check for empty parent (and self-href) diff --git a/libgrive/src/base/Resource.hh b/libgrive/src/base/Resource.hh index e7ee0ac3..57d7319c 100644 --- a/libgrive/src/base/Resource.hh +++ b/libgrive/src/base/Resource.hh @@ -19,6 +19,7 @@ #pragma once +#include "util/Types.hh" #include "util/DateTime.hh" #include "util/Exception.hh" #include "util/FileSystem.hh" @@ -108,7 +109,9 @@ public : bool IsInRootTree() const ; bool IsRoot() const ; bool HasID() const ; + u64_t Size() const; std::string MD5() const ; + std::string GetMD5() ; void FromRemote( const Entry& remote ) ; void FromDeleted( Val& state ) ; @@ -141,6 +144,7 @@ private : void DeleteIndex() ; void SetIndex( bool ) ; + bool CheckRename( Syncer* syncer, ResourceTree *res_tree ) ; void SyncSelf( Syncer* syncer, ResourceTree *res_tree, const Val& options ) ; private : @@ -149,6 +153,7 @@ private : std::string m_md5 ; DateTime m_mtime ; DateTime m_ctime ; + u64_t m_size ; std::string m_id ; std::string m_href ; diff --git a/libgrive/src/base/ResourceTree.cc b/libgrive/src/base/ResourceTree.cc index 16ac5347..e6eb6190 100644 --- a/libgrive/src/base/ResourceTree.cc +++ b/libgrive/src/base/ResourceTree.cc @@ -105,7 +105,13 @@ MD5Range ResourceTree::FindByMD5( const std::string& md5 ) return MD5Range( map.end(), map.end() ) ; } -/// Reinsert should be called when the ID/HREF were updated +SizeRange ResourceTree::FindBySize( u64_t size ) +{ + SizeMap& map = m_set.get() ; + return map.equal_range( size ); +} + +/// Reinsert should be called when the ID/HREF/MD5 were updated bool ResourceTree::ReInsert( Resource *coll ) { Set& s = m_set.get() ; diff --git a/libgrive/src/base/ResourceTree.hh b/libgrive/src/base/ResourceTree.hh index b1fc7285..b28c4a3b 100644 --- a/libgrive/src/base/ResourceTree.hh +++ b/libgrive/src/base/ResourceTree.hh @@ -36,19 +36,23 @@ namespace details struct ByMD5 {} ; struct ByHref {} ; struct ByIdentity {} ; + struct BySize {} ; typedef multi_index_container< Resource*, indexed_by< hashed_non_unique, const_mem_fun >, hashed_non_unique, const_mem_fun >, + hashed_non_unique, const_mem_fun >, hashed_unique, identity > > > Folders ; typedef Folders::index::type MD5Map ; typedef Folders::index::type HrefMap ; + typedef Folders::index::type SizeMap ; typedef Folders::index::type Set ; + typedef std::pair SizeRange ; typedef std::pair MD5Range ; } @@ -70,6 +74,7 @@ public : Resource* FindByHref( const std::string& href ) ; const Resource* FindByHref( const std::string& href ) const ; details::MD5Range FindByMD5( const std::string& md5 ) ; + details::SizeRange FindBySize( u64_t size ) ; bool ReInsert( Resource *coll ) ; diff --git a/libgrive/src/base/State.cc b/libgrive/src/base/State.cc index 44d8bf91..9fddf4d1 100644 --- a/libgrive/src/base/State.cc +++ b/libgrive/src/base/State.cc @@ -104,20 +104,21 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) { // if the Resource object of the child already exists, it should // have been so no need to do anything here - Resource *c = folder->FindChild( fname ) ; + Resource *c = folder->FindChild( fname ), *c2 = c ; if ( !c ) { - c = new Resource( fname, "" ) ; - folder->AddChild( c ) ; - m_res.Insert( c ) ; + c2 = new Resource( fname, "" ) ; + folder->AddChild( c2 ) ; } leftover.erase( fname ); Val& rec = tree.Item( fname ); if ( m_force ) rec.Del( "srv_time" ); - c->FromLocal( rec ) ; - if ( c->IsFolder() ) - FromLocal( *i, c, rec.Item( "tree" ) ) ; + c2->FromLocal( rec ) ; + if ( !c ) + m_res.Insert( c2 ) ; + if ( c2->IsFolder() ) + FromLocal( *i, c2, rec.Item( "tree" ) ) ; } } @@ -129,17 +130,18 @@ void State::FromLocal( const fs::path& p, Resource* folder, Val& tree ) else { // Restore state of locally deleted files - Resource *c = folder->FindChild( i->first ) ; + Resource *c = folder->FindChild( i->first ), *c2 ; if ( !c ) { - c = new Resource( i->first, i->second.Has( "tree" ) ? "folder" : "file" ) ; - folder->AddChild( c ) ; - m_res.Insert( c ) ; + c2 = new Resource( i->first, i->second.Has( "tree" ) ? "folder" : "file" ) ; + folder->AddChild( c2 ) ; } Val& rec = tree.Item( i->first ); if ( m_force || m_ign_changed ) rec.Del( "srv_time" ); - c->FromDeleted( rec ); + c2->FromDeleted( rec ); + if ( !c ) + m_res.Insert( c2 ) ; } } } diff --git a/libgrive/src/drive2/Entry2.cc b/libgrive/src/drive2/Entry2.cc index 75f7d96f..b445c1b8 100644 --- a/libgrive/src/drive2/Entry2.cc +++ b/libgrive/src/drive2/Entry2.cc @@ -44,6 +44,7 @@ void Entry2::Update( const Val& item ) // changestamp only appears in change feed entries m_change_stamp = is_chg ? item["id"].Int() : -1 ; m_is_removed = is_chg && item["deleted"].Bool() ; + m_size = 0 ; const Val& file = is_chg && !m_is_removed ? item["file"] : item; @@ -75,6 +76,7 @@ void Entry2::Update( const Val& item ) else { m_md5 = file["md5Checksum"] ; + m_size = file["fileSize"].U64() ; m_content_src = file["downloadUrl"] ; // convert to lower case for easy comparison std::transform( m_md5.begin(), m_md5.end(), m_md5.begin(), tolower ) ; diff --git a/libgrive/src/util/OS.cc b/libgrive/src/util/OS.cc index 3569aa15..9095e730 100644 --- a/libgrive/src/util/OS.cc +++ b/libgrive/src/util/OS.cc @@ -44,7 +44,7 @@ void Stat( const fs::path& filename, DateTime *t, off_t *size, bool *is_dir ) Stat( filename.string(), t, size, is_dir ) ; } -void Stat( const std::string& filename, DateTime *t, off_t *size, bool *is_dir ) +void Stat( const std::string& filename, DateTime *t, off64_t *size, bool *is_dir ) { struct stat s = {} ; if ( ::stat( filename.c_str(), &s ) != 0 ) @@ -65,10 +65,10 @@ void Stat( const std::string& filename, DateTime *t, off_t *size, bool *is_dir ) *t = DateTime( s.st_ctim.tv_sec, s.st_ctim.tv_nsec); #endif } - if (size) + if ( size ) *size = s.st_size; - if (is_dir) - *is_dir = S_ISDIR(s.st_mode) ? true : false; + if ( is_dir ) + *is_dir = S_ISDIR( s.st_mode ) ? true : false; } void SetFileTime( const fs::path& filename, const DateTime& t ) diff --git a/libgrive/src/util/OS.hh b/libgrive/src/util/OS.hh index edb9a12c..43497839 100644 --- a/libgrive/src/util/OS.hh +++ b/libgrive/src/util/OS.hh @@ -33,8 +33,8 @@ namespace os { struct Error : virtual Exception {} ; - void Stat( const std::string& filename, DateTime *t, off_t *size, bool *is_dir ) ; - void Stat( const fs::path& filename, DateTime *t, off_t *size, bool *is_dir ) ; + void Stat( const std::string& filename, DateTime *t, off64_t *size, bool *is_dir ) ; + void Stat( const fs::path& filename, DateTime *t, off64_t *size, bool *is_dir ) ; void SetFileTime( const std::string& filename, const DateTime& t ) ; void SetFileTime( const fs::path& filename, const DateTime& t ) ; From 11a3d788d014d310fe5b725a41a4143a02530537 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Wed, 28 Sep 2016 10:16:57 +0300 Subject: [PATCH 157/166] Fix libgrive/test build after removing doclist api (fixes #102) --- libgrive/test/UnitTest.cc | 6 ++---- libgrive/test/base/ResourceTest.cc | 13 ++++++------- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/libgrive/test/UnitTest.cc b/libgrive/test/UnitTest.cc index 7dfbf772..02121d21 100644 --- a/libgrive/test/UnitTest.cc +++ b/libgrive/test/UnitTest.cc @@ -21,7 +21,6 @@ #include "util/log/DefaultLog.hh" -#include "drive/EntryTest.hh" #include "base/ResourceTest.hh" #include "base/ResourceTreeTest.hh" #include "base/StateTest.hh" @@ -29,7 +28,7 @@ #include "util/FunctionTest.hh" #include "util/ConfigTest.hh" #include "util/SignalHandlerTest.hh" -#include "xml/NodeTest.hh" +//#include "xml/NodeTest.hh" int main( int argc, char **argv ) { @@ -38,7 +37,6 @@ int main( int argc, char **argv ) gr::LogBase::Inst( new gr::log::DefaultLog ) ; CppUnit::TextUi::TestRunner runner; - runner.addTest( Entry1Test::suite( ) ) ; runner.addTest( StateTest::suite( ) ) ; runner.addTest( ResourceTest::suite( ) ) ; runner.addTest( ResourceTreeTest::suite( ) ) ; @@ -46,7 +44,7 @@ int main( int argc, char **argv ) runner.addTest( FunctionTest::suite( ) ) ; runner.addTest( ConfigTest::suite( ) ) ; runner.addTest( SignalHandlerTest::suite( ) ) ; - runner.addTest( NodeTest::suite( ) ) ; + //runner.addTest( NodeTest::suite( ) ) ; runner.run(); return 0 ; diff --git a/libgrive/test/base/ResourceTest.cc b/libgrive/test/base/ResourceTest.cc index f40dc38b..f22a9a53 100644 --- a/libgrive/test/base/ResourceTest.cc +++ b/libgrive/test/base/ResourceTest.cc @@ -23,8 +23,7 @@ #include "base/Resource.hh" -#include "drive/Entry1.hh" -#include "xml/Node.hh" +#include "drive2/Entry2.hh" #include "json/Val.hh" #include @@ -32,7 +31,7 @@ namespace grut { using namespace gr ; -using namespace gr::v1 ; +using namespace gr::v2 ; ResourceTest::ResourceTest( ) { @@ -61,11 +60,11 @@ void ResourceTest::TestNormal( ) GRUT_ASSERT_EQUAL( subject.MD5(), "c0742c0a32b2c909b6f176d17a6992d0" ) ; GRUT_ASSERT_EQUAL( subject.StateStr(), "local_new" ) ; - xml::Node entry = xml::Node::Element( "entry" ) ; - entry.AddElement( "updated" ).AddText( "2012-05-09T16:13:22.401Z" ) ; - entry.AddElement( "docs:md5Checksum" ).AddText( "DIFFERENT" ) ; + Val entry; + entry.Set( "modifiedDate", Val( std::string( "2012-05-09T16:13:22.401Z" ) ) ); + entry.Set( "md5Checksum", Val( std::string( "DIFFERENT" ) ) ); - Entry1 remote( entry ) ; + Entry2 remote( entry ) ; GRUT_ASSERT_EQUAL( "different", remote.MD5() ) ; subject.FromRemote( remote ) ; GRUT_ASSERT_EQUAL( "local_changed", subject.StateStr() ) ; From f27e3724dedf2d1ab0e19586d97f4df37ff3bdab Mon Sep 17 00:00:00 2001 From: Blackrabbit Date: Fri, 5 Aug 2016 19:29:48 +0200 Subject: [PATCH 158/166] Implement progressbar (by @svartkanin adjusted by @vitalif) --- README.md | 3 +- grive/doc/grive.1 | 3 ++ grive/src/main.cc | 15 ++++++ libgrive/src/base/Resource.cc | 3 +- libgrive/src/base/Syncer.cc | 2 +- libgrive/src/drive2/Feed2.cc | 2 +- libgrive/src/drive2/Syncer2.cc | 2 +- libgrive/src/http/Agent.cc | 5 +- libgrive/src/http/Agent.hh | 9 +++- libgrive/src/http/CurlAgent.cc | 34 ++++++++++-- libgrive/src/http/CurlAgent.hh | 13 +++-- libgrive/src/protocol/AuthAgent.cc | 10 +++- libgrive/src/protocol/AuthAgent.hh | 5 +- libgrive/src/util/Progress.hh | 14 +++++ libgrive/src/util/ProgressBar.cc | 86 ++++++++++++++++++++++++++++++ libgrive/src/util/ProgressBar.hh | 25 +++++++++ 16 files changed, 211 insertions(+), 20 deletions(-) create mode 100644 libgrive/src/util/Progress.hh create mode 100644 libgrive/src/util/ProgressBar.cc create mode 100644 libgrive/src/util/ProgressBar.hh diff --git a/README.md b/README.md index 89bfec19..2e2ee468 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Grive2 0.5.1-dev -14 Jan 2016, Vitaliy Filippov +28 Sep 2016, Vitaliy Filippov http://yourcmc.ru/wiki/Grive2 @@ -91,6 +91,7 @@ Grive uses cmake to build. Basic install sequence is - added options to limit upload and download speed - faster upload of new and changed files. now Grive uploads files without first calculating md5 checksum when file is created locally or when its size changes. +- added -P/--progress-bar option to print ASCII progress bar for each processed file (pull request by @svartkanin) ### Grive2 v0.5 diff --git a/grive/doc/grive.1 b/grive/doc/grive.1 index 3b0b56ad..cf0bc26d 100644 --- a/grive/doc/grive.1 +++ b/grive/doc/grive.1 @@ -75,6 +75,9 @@ subdirectory. Internally converted to an ignore regexp, remembered for next runs \fB\-v\fR, \fB\-\-version\fR Displays program version .TP +\fB\-P\fR, \fB\-\-progress-bar\fR +Print ASCII progress bar for each downloaded/uploaded file. +.TP \fB\-V\fR, \fB\-\-verbose\fR Verbose mode. Enables more messages than usual. diff --git a/grive/src/main.cc b/grive/src/main.cc index eef90d37..c87a8543 100644 --- a/grive/src/main.cc +++ b/grive/src/main.cc @@ -18,6 +18,7 @@ */ #include "util/Config.hh" +#include "util/ProgressBar.hh" #include "base/Drive.hh" #include "drive2/Syncer2.hh" @@ -126,6 +127,7 @@ int Main( int argc, char **argv ) ( "ignore", po::value(), "Perl RegExp to ignore files (matched against relative paths)." ) ( "upload-speed,U", po::value(), "Limit upload speed in kbytes per second" ) ( "download-speed,D", po::value(), "Limit download speed in kbytes per second" ) + ( "progress-bar,P", "Enable progress bar for upload/download of files") ; po::variables_map vm; @@ -156,6 +158,13 @@ int Main( int argc, char **argv ) if ( vm.count( "log-http" ) ) http->SetLog( new http::ResponseLog( vm["log-http"].as(), ".txt" ) ); + std::unique_ptr pb; + if ( vm.count( "progress-bar" ) ) + { + pb.reset( new ProgressBar() ); + http->SetProgressReporter( pb.get() ); + } + if ( vm.count( "auth" ) ) { OAuth2 token( http.get(), client_id, client_secret ) ; @@ -208,7 +217,13 @@ int Main( int argc, char **argv ) if ( vm.count( "dry-run" ) == 0 ) { + // The progress bar should just be enabled when actual file transfers take place + if ( pb ) + pb->setShowProgressBar( true ) ; drive.Update() ; + if ( pb ) + pb->setShowProgressBar( false ) ; + drive.SaveState() ; } else diff --git a/libgrive/src/base/Resource.cc b/libgrive/src/base/Resource.cc index b03aa5c9..23ce335d 100644 --- a/libgrive/src/base/Resource.cc +++ b/libgrive/src/base/Resource.cc @@ -187,7 +187,7 @@ void Resource::FromRemoteFile( const Entry& remote ) { Log( "file %1% is created in remote (change %2%)", path, remote.ChangeStamp(), log::verbose ) ; - + m_size = remote.Size(); m_state = remote_new ; } else @@ -214,6 +214,7 @@ void Resource::FromRemoteFile( const Entry& remote ) if ( remote.MTime().Sec() > m_mtime.Sec() ) { Log( "file %1% is changed in remote", path, log::verbose ) ; + m_size = remote.Size(); m_state = remote_changed ; } diff --git a/libgrive/src/base/Syncer.cc b/libgrive/src/base/Syncer.cc index 2cc312c0..f68d3090 100644 --- a/libgrive/src/base/Syncer.cc +++ b/libgrive/src/base/Syncer.cc @@ -41,7 +41,7 @@ http::Agent* Syncer::Agent() const void Syncer::Download( Resource *res, const fs::path& file ) { http::Download dl( file.string(), http::Download::NoChecksum() ) ; - long r = m_http->Get( res->ContentSrc(), &dl, http::Header() ) ; + long r = m_http->Get( res->ContentSrc(), &dl, http::Header(), res->Size() ) ; if ( r <= 400 ) { if ( res->ServerTime() != DateTime() ) diff --git a/libgrive/src/drive2/Feed2.cc b/libgrive/src/drive2/Feed2.cc index 8a3244ee..8f3d2344 100644 --- a/libgrive/src/drive2/Feed2.cc +++ b/libgrive/src/drive2/Feed2.cc @@ -42,7 +42,7 @@ bool Feed2::GetNext( http::Agent *http ) return false ; http::ValResponse out ; - http->Get( m_next, &out, http::Header() ) ; + http->Get( m_next, &out, http::Header(), 0 ) ; Val m_content = out.Response() ; Val::Array items = m_content["items"].AsArray() ; diff --git a/libgrive/src/drive2/Syncer2.cc b/libgrive/src/drive2/Syncer2.cc index f7c21d2e..fa5e05dd 100644 --- a/libgrive/src/drive2/Syncer2.cc +++ b/libgrive/src/drive2/Syncer2.cc @@ -235,7 +235,7 @@ std::unique_ptr Syncer2::GetChanges( long min_cstamp ) long Syncer2::GetChangeStamp( long min_cstamp ) { http::ValResponse res ; - m_http->Get( ChangesFeed( min_cstamp, 1 ), &res, http::Header() ) ; + m_http->Get( ChangesFeed( min_cstamp, 1 ), &res, http::Header(), 0 ) ; return std::atoi( res.Response()["largestChangeId"].Str().c_str() ); } diff --git a/libgrive/src/http/Agent.cc b/libgrive/src/http/Agent.cc index 39276ac0..7cfc99fa 100644 --- a/libgrive/src/http/Agent.cc +++ b/libgrive/src/http/Agent.cc @@ -52,9 +52,10 @@ long Agent::Put( long Agent::Get( const std::string& url, DataStream *dest, - const Header& hdr ) + const Header& hdr, + u64_t downloadFileBytes ) { - return Request( "GET", url, NULL, dest, hdr ); + return Request( "GET", url, NULL, dest, hdr, downloadFileBytes ); } long Agent::Post( diff --git a/libgrive/src/http/Agent.hh b/libgrive/src/http/Agent.hh index 8268e197..3ca7c118 100644 --- a/libgrive/src/http/Agent.hh +++ b/libgrive/src/http/Agent.hh @@ -22,6 +22,7 @@ #include #include "ResponseLog.hh" #include "util/Types.hh" +#include "util/Progress.hh" namespace gr { @@ -59,7 +60,8 @@ public : virtual long Get( const std::string& url, DataStream *dest, - const Header& hdr ) ; + const Header& hdr, + u64_t downloadFileBytes = 0 ) ; virtual long Post( const std::string& url, @@ -72,7 +74,8 @@ public : const std::string& url, SeekStream *in, DataStream *dest, - const Header& hdr ) = 0 ; + const Header& hdr, + u64_t downloadFileBytes = 0 ) = 0 ; virtual void SetUploadSpeed( unsigned kbytes ) ; virtual void SetDownloadSpeed( unsigned kbytes ) ; @@ -84,6 +87,8 @@ public : virtual std::string Escape( const std::string& str ) = 0 ; virtual std::string Unescape( const std::string& str ) = 0 ; + + virtual void SetProgressReporter( Progress* ) = 0; } ; } } // end of namespace diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 3bfdfca9..122822d3 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -28,9 +28,6 @@ #include -// dependent libraries -#include - #include #include #include @@ -68,6 +65,7 @@ struct CurlAgent::Impl std::string error_headers ; std::string error_data ; DataStream *dest ; + u64_t total_download, total_upload ; } ; static struct curl_slist* SetHeader( CURL* handle, const Header& hdr ); @@ -94,6 +92,7 @@ void CurlAgent::Init() m_pimpl->error_headers = ""; m_pimpl->error_data = ""; m_pimpl->dest = NULL; + m_pimpl->total_download = m_pimpl->total_upload = 0; } CurlAgent::~CurlAgent() @@ -111,6 +110,11 @@ void CurlAgent::SetLog(ResponseLog *log) m_log.reset( log ); } +void CurlAgent::SetProgressReporter(Progress *progress) +{ + m_pb = progress; +} + std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) { char *str = static_cast(ptr) ; @@ -131,7 +135,7 @@ std::size_t CurlAgent::HeaderCallback( void *ptr, size_t size, size_t nmemb, Cur if ( pos != line.npos ) { std::size_t end_pos = line.find( "\r\n", pos ) ; - pthis->m_pimpl->location = line.substr( loc.size(), end_pos - loc.size() ) ; + pthis->m_pimpl->location = line.substr( pos+loc.size(), end_pos - loc.size() ) ; } return size*nmemb ; @@ -142,6 +146,7 @@ std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, CurlAgent assert( pthis != 0 ) ; if ( pthis->m_log.get() ) pthis->m_log->Write( (const char*)ptr, size*nmemb ); + if ( pthis->m_pimpl->error && pthis->m_pimpl->error_data.size() < 65536 ) { // Do not feed error responses to destination stream @@ -151,6 +156,19 @@ std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, CurlAgent return pthis->m_pimpl->dest->Write( static_cast(ptr), size * nmemb ) ; } +int CurlAgent::progress_callback( CurlAgent *pthis, curl_off_t totalDownload, curl_off_t finishedDownload, curl_off_t totalUpload, curl_off_t finishedUpload ) +{ + // Only report download progress when set explicitly + totalDownload = pthis->m_pimpl->total_download; + if ( !totalUpload ) + totalUpload = pthis->m_pimpl->total_upload; + pthis->m_pb->reportProgress( + totalDownload > 0 ? totalDownload : totalUpload, + totalDownload > 0 ? finishedDownload : finishedUpload + ); + return 0; +} + long CurlAgent::ExecCurl( const std::string& url, DataStream *dest, @@ -168,6 +186,10 @@ long CurlAgent::ExecCurl( struct curl_slist *slist = SetHeader( m_pimpl->curl, hdr ) ; + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); + curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progress_callback); + curl_easy_setopt(curl, CURLOPT_XFERINFODATA, this); + CURLcode curl_code = ::curl_easy_perform(curl); curl_slist_free_all(slist); @@ -202,11 +224,13 @@ long CurlAgent::Request( const std::string& url, SeekStream *in, DataStream *dest, - const Header& hdr ) + const Header& hdr, + u64_t downloadFileBytes ) { Trace("HTTP %1% \"%2%\"", method, url ) ; Init() ; + m_pimpl->total_download = downloadFileBytes ; CURL *curl = m_pimpl->curl ; // set common options diff --git a/libgrive/src/http/CurlAgent.hh b/libgrive/src/http/CurlAgent.hh index c25fa1a0..b659f501 100644 --- a/libgrive/src/http/CurlAgent.hh +++ b/libgrive/src/http/CurlAgent.hh @@ -24,6 +24,8 @@ #include #include +#include + namespace gr { class DataStream ; @@ -43,13 +45,15 @@ public : ResponseLog* GetLog() const ; void SetLog( ResponseLog *log ) ; + void SetProgressReporter( Progress *progress ) ; long Request( const std::string& method, const std::string& url, SeekStream *in, DataStream *dest, - const Header& hdr ) ; + const Header& hdr, + u64_t downloadFileBytes = 0 ) ; std::string LastError() const ; std::string LastErrorHeaders() const ; @@ -59,6 +63,8 @@ public : std::string Escape( const std::string& str ) ; std::string Unescape( const std::string& str ) ; + static int progress_callback( CurlAgent *pthis, curl_off_t totalDownload, curl_off_t finishedDownload, curl_off_t totalUpload, curl_off_t finishedUpload ); + private : static std::size_t HeaderCallback( void *ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; static std::size_t Receive( void* ptr, size_t size, size_t nmemb, CurlAgent *pthis ) ; @@ -72,8 +78,9 @@ private : private : struct Impl ; - std::unique_ptr m_pimpl ; - std::unique_ptr m_log ; + std::unique_ptr m_pimpl ; + std::unique_ptr m_log ; + Progress* m_pb ; } ; } } // end of namespace diff --git a/libgrive/src/protocol/AuthAgent.cc b/libgrive/src/protocol/AuthAgent.cc index f2f59d6f..6baf1133 100644 --- a/libgrive/src/protocol/AuthAgent.cc +++ b/libgrive/src/protocol/AuthAgent.cc @@ -48,6 +48,11 @@ void AuthAgent::SetLog( http::ResponseLog *log ) return m_agent->SetLog( log ); } +void AuthAgent::SetProgressReporter( Progress *progress ) +{ + m_agent->SetProgressReporter( progress ); +} + void AuthAgent::SetUploadSpeed( unsigned kbytes ) { m_agent->SetUploadSpeed( kbytes ); @@ -71,7 +76,8 @@ long AuthAgent::Request( const std::string& url, SeekStream *in, DataStream *dest, - const http::Header& hdr ) + const http::Header& hdr, + u64_t downloadFileBytes ) { long response; Header auth; @@ -80,7 +86,7 @@ long AuthAgent::Request( auth = AppendHeader( hdr ); if ( in ) in->Seek( 0, 0 ); - response = m_agent->Request( method, url, in, dest, auth ); + response = m_agent->Request( method, url, in, dest, auth, downloadFileBytes ); } while ( CheckRetry( response ) ); return CheckHttpResponse( response, url, auth ); } diff --git a/libgrive/src/protocol/AuthAgent.hh b/libgrive/src/protocol/AuthAgent.hh index 7f16dc14..328e8cf4 100644 --- a/libgrive/src/protocol/AuthAgent.hh +++ b/libgrive/src/protocol/AuthAgent.hh @@ -44,7 +44,8 @@ public : const std::string& url, SeekStream *in, DataStream *dest, - const http::Header& hdr ) ; + const http::Header& hdr, + u64_t downloadFileBytes = 0 ) ; std::string LastError() const ; std::string LastErrorHeaders() const ; @@ -57,6 +58,8 @@ public : void SetUploadSpeed( unsigned kbytes ) ; void SetDownloadSpeed( unsigned kbytes ) ; + void SetProgressReporter( Progress *progress ) ; + private : http::Header AppendHeader( const http::Header& hdr ) const ; bool CheckRetry( long response ) ; diff --git a/libgrive/src/util/Progress.hh b/libgrive/src/util/Progress.hh new file mode 100644 index 00000000..9e087f90 --- /dev/null +++ b/libgrive/src/util/Progress.hh @@ -0,0 +1,14 @@ +#pragma once + +#include "util/Types.hh" + +namespace gr { + +class Progress +{ +public: + virtual void reportProgress(u64_t total, u64_t processed) = 0; +}; + +} +; diff --git a/libgrive/src/util/ProgressBar.cc b/libgrive/src/util/ProgressBar.cc new file mode 100644 index 00000000..9ad8de77 --- /dev/null +++ b/libgrive/src/util/ProgressBar.cc @@ -0,0 +1,86 @@ +#include +#include +#include +#include + +#include "ProgressBar.hh" + +namespace gr +{ + +ProgressBar::ProgressBar(): showProgressBar(false), last(1000) +{ +} + +ProgressBar::~ProgressBar() +{ +} + +void ProgressBar::setShowProgressBar(bool showProgressBar) +{ + this->showProgressBar = showProgressBar; +} + +unsigned short int ProgressBar::determineTerminalSize() +{ + struct winsize w; + ioctl(STDOUT_FILENO, TIOCGWINSZ, &w); + return w.ws_col; +} + +void ProgressBar::printBytes(u64_t bytes) +{ + if (bytes >= 1024*1024*1024) + printf("%.3f GB", (double)bytes/1024/1024/1024); + else if (bytes >= 1024*1024) + printf("%.3f MB", (double)bytes/1024/1024); + else + printf("%.3f KB", (double)bytes/1024); +} + +void ProgressBar::reportProgress(u64_t total, u64_t processed) +{ + if (showProgressBar && total) + { + // libcurl seems to process more bytes then the actual file size :) + if (processed > total) + processed = total; + double fraction = (double)processed/total; + + int point = round(fraction*1000); + if (point != this->last) + { + // only print progress after >= 0.1% change + this->last = point; + + // 10 for prefix of percent and 22 for suffix of file size + int availableSize = determineTerminalSize() - 32; + int totalDots; + if (availableSize > 100) + totalDots = 100; + else if (availableSize < 0) + totalDots = 10; + else + totalDots = availableSize; + + int dotz = round(fraction * totalDots); + int count = 0; + // delete previous output line + printf("\r\33[2K [%3.0f%%] [", fraction * 100); + for (; count < dotz - 1; count++) + putchar('='); + putchar('>'); + for (; count < totalDots - 1; count++) + putchar(' '); + printf("] "); + printBytes(processed); + putchar('/'); + printBytes(total); + if (point == 1000) + putchar('\n'); + fflush(stdout); + } + } +} + +} diff --git a/libgrive/src/util/ProgressBar.hh b/libgrive/src/util/ProgressBar.hh new file mode 100644 index 00000000..5e10eeea --- /dev/null +++ b/libgrive/src/util/ProgressBar.hh @@ -0,0 +1,25 @@ +#pragma once + +#include "util/Progress.hh" + +namespace gr { + +class ProgressBar: public Progress +{ +public: + ProgressBar(); + virtual ~ProgressBar(); + + void reportProgress(u64_t total, u64_t processed); + void setShowProgressBar(bool showProgressBar); + +private: + static void printBytes(u64_t bytes); + static unsigned short int determineTerminalSize(); + + bool showProgressBar; + int last; +}; + +} +; From 67b5b05e1795347db67a1547bef6ca77d98732a5 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 30 Sep 2016 01:14:50 +0300 Subject: [PATCH 159/166] Fix auth segfault, remove 1/1000 progressbar interval (fixes #103) --- libgrive/src/http/CurlAgent.cc | 17 ++++++---- libgrive/src/util/ProgressBar.cc | 57 +++++++++++++++----------------- libgrive/src/util/ProgressBar.hh | 1 - 3 files changed, 36 insertions(+), 39 deletions(-) diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 122822d3..2e1f46d7 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -159,13 +159,16 @@ std::size_t CurlAgent::Receive( void* ptr, size_t size, size_t nmemb, CurlAgent int CurlAgent::progress_callback( CurlAgent *pthis, curl_off_t totalDownload, curl_off_t finishedDownload, curl_off_t totalUpload, curl_off_t finishedUpload ) { // Only report download progress when set explicitly - totalDownload = pthis->m_pimpl->total_download; - if ( !totalUpload ) - totalUpload = pthis->m_pimpl->total_upload; - pthis->m_pb->reportProgress( - totalDownload > 0 ? totalDownload : totalUpload, - totalDownload > 0 ? finishedDownload : finishedUpload - ); + if ( pthis->m_pb ) + { + totalDownload = pthis->m_pimpl->total_download; + if ( !totalUpload ) + totalUpload = pthis->m_pimpl->total_upload; + pthis->m_pb->reportProgress( + totalDownload > 0 ? totalDownload : totalUpload, + totalDownload > 0 ? finishedDownload : finishedUpload + ); + } return 0; } diff --git a/libgrive/src/util/ProgressBar.cc b/libgrive/src/util/ProgressBar.cc index 9ad8de77..d36e0a44 100644 --- a/libgrive/src/util/ProgressBar.cc +++ b/libgrive/src/util/ProgressBar.cc @@ -8,7 +8,7 @@ namespace gr { -ProgressBar::ProgressBar(): showProgressBar(false), last(1000) +ProgressBar::ProgressBar(): showProgressBar(false) { } @@ -48,38 +48,33 @@ void ProgressBar::reportProgress(u64_t total, u64_t processed) double fraction = (double)processed/total; int point = round(fraction*1000); - if (point != this->last) - { - // only print progress after >= 0.1% change - this->last = point; - // 10 for prefix of percent and 22 for suffix of file size - int availableSize = determineTerminalSize() - 32; - int totalDots; - if (availableSize > 100) - totalDots = 100; - else if (availableSize < 0) - totalDots = 10; - else - totalDots = availableSize; + // 10 for prefix of percent and 22 for suffix of file size + int availableSize = determineTerminalSize() - 32; + int totalDots; + if (availableSize > 100) + totalDots = 100; + else if (availableSize < 0) + totalDots = 10; + else + totalDots = availableSize; - int dotz = round(fraction * totalDots); - int count = 0; - // delete previous output line - printf("\r\33[2K [%3.0f%%] [", fraction * 100); - for (; count < dotz - 1; count++) - putchar('='); - putchar('>'); - for (; count < totalDots - 1; count++) - putchar(' '); - printf("] "); - printBytes(processed); - putchar('/'); - printBytes(total); - if (point == 1000) - putchar('\n'); - fflush(stdout); - } + int dotz = round(fraction * totalDots); + int count = 0; + // delete previous output line + printf("\r\33[2K [%3.0f%%] [", fraction * 100); + for (; count < dotz - 1; count++) + putchar('='); + putchar('>'); + for (; count < totalDots - 1; count++) + putchar(' '); + printf("] "); + printBytes(processed); + putchar('/'); + printBytes(total); + if (point == 1000) + putchar('\n'); + fflush(stdout); } } diff --git a/libgrive/src/util/ProgressBar.hh b/libgrive/src/util/ProgressBar.hh index 5e10eeea..fbad6af9 100644 --- a/libgrive/src/util/ProgressBar.hh +++ b/libgrive/src/util/ProgressBar.hh @@ -18,7 +18,6 @@ private: static unsigned short int determineTerminalSize(); bool showProgressBar; - int last; }; } From 68e0e5afe59ddb41ad5fb6ee97a2ad2e8bf8f67d Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Fri, 30 Sep 2016 12:33:11 +0300 Subject: [PATCH 160/166] Get progressbar::last back, do not print multiple 100% lines (fixes #104) --- libgrive/src/util/ProgressBar.cc | 57 +++++++++++++++++--------------- libgrive/src/util/ProgressBar.hh | 1 + 2 files changed, 32 insertions(+), 26 deletions(-) diff --git a/libgrive/src/util/ProgressBar.cc b/libgrive/src/util/ProgressBar.cc index d36e0a44..60417013 100644 --- a/libgrive/src/util/ProgressBar.cc +++ b/libgrive/src/util/ProgressBar.cc @@ -8,7 +8,7 @@ namespace gr { -ProgressBar::ProgressBar(): showProgressBar(false) +ProgressBar::ProgressBar(): showProgressBar(false), last(1000) { } @@ -48,33 +48,38 @@ void ProgressBar::reportProgress(u64_t total, u64_t processed) double fraction = (double)processed/total; int point = round(fraction*1000); + if (this->last < 1000 || point != this->last) + { + // only print progress after >= 0.1% change + this->last = point; - // 10 for prefix of percent and 22 for suffix of file size - int availableSize = determineTerminalSize() - 32; - int totalDots; - if (availableSize > 100) - totalDots = 100; - else if (availableSize < 0) - totalDots = 10; - else - totalDots = availableSize; + // 10 for prefix of percent and 22 for suffix of file size + int availableSize = determineTerminalSize() - 32; + int totalDots; + if (availableSize > 100) + totalDots = 100; + else if (availableSize < 0) + totalDots = 10; + else + totalDots = availableSize; - int dotz = round(fraction * totalDots); - int count = 0; - // delete previous output line - printf("\r\33[2K [%3.0f%%] [", fraction * 100); - for (; count < dotz - 1; count++) - putchar('='); - putchar('>'); - for (; count < totalDots - 1; count++) - putchar(' '); - printf("] "); - printBytes(processed); - putchar('/'); - printBytes(total); - if (point == 1000) - putchar('\n'); - fflush(stdout); + int dotz = round(fraction * totalDots); + int count = 0; + // delete previous output line + printf("\r\33[2K [%3.0f%%] [", fraction * 100); + for (; count < dotz - 1; count++) + putchar('='); + putchar('>'); + for (; count < totalDots - 1; count++) + putchar(' '); + printf("] "); + printBytes(processed); + putchar('/'); + printBytes(total); + if (point == 1000) + putchar('\n'); + fflush(stdout); + } } } diff --git a/libgrive/src/util/ProgressBar.hh b/libgrive/src/util/ProgressBar.hh index fbad6af9..5e10eeea 100644 --- a/libgrive/src/util/ProgressBar.hh +++ b/libgrive/src/util/ProgressBar.hh @@ -18,6 +18,7 @@ private: static unsigned short int determineTerminalSize(); bool showProgressBar; + int last; }; } From e3f948496d8cb6ab7a4d706af483b9a1b749adae Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 1 Oct 2016 01:49:30 +0300 Subject: [PATCH 161/166] Fix #105 --- libgrive/src/util/ProgressBar.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libgrive/src/util/ProgressBar.cc b/libgrive/src/util/ProgressBar.cc index 60417013..b21aedd1 100644 --- a/libgrive/src/util/ProgressBar.cc +++ b/libgrive/src/util/ProgressBar.cc @@ -53,8 +53,8 @@ void ProgressBar::reportProgress(u64_t total, u64_t processed) // only print progress after >= 0.1% change this->last = point; - // 10 for prefix of percent and 22 for suffix of file size - int availableSize = determineTerminalSize() - 32; + // 10 for prefix of percent and 23 for suffix of file size + int availableSize = determineTerminalSize() - 33; int totalDots; if (availableSize > 100) totalDots = 100; From 08e29070c2352b90260eb769ead5cf5673d50fb9 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 1 Oct 2016 02:04:04 +0300 Subject: [PATCH 162/166] Fix possible uninit m_pb, print \33[K (clreol) in the end for less glitches in xfce4-terminal and \r for better window resize --- libgrive/src/http/CurlAgent.cc | 2 +- libgrive/src/util/ProgressBar.cc | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/libgrive/src/http/CurlAgent.cc b/libgrive/src/http/CurlAgent.cc index 2e1f46d7..eee9a8d9 100644 --- a/libgrive/src/http/CurlAgent.cc +++ b/libgrive/src/http/CurlAgent.cc @@ -71,7 +71,7 @@ struct CurlAgent::Impl static struct curl_slist* SetHeader( CURL* handle, const Header& hdr ); CurlAgent::CurlAgent() : Agent(), - m_pimpl( new Impl ) + m_pimpl( new Impl ), m_pb( 0 ) { m_pimpl->curl = ::curl_easy_init(); } diff --git a/libgrive/src/util/ProgressBar.cc b/libgrive/src/util/ProgressBar.cc index b21aedd1..efd71bcf 100644 --- a/libgrive/src/util/ProgressBar.cc +++ b/libgrive/src/util/ProgressBar.cc @@ -66,7 +66,7 @@ void ProgressBar::reportProgress(u64_t total, u64_t processed) int dotz = round(fraction * totalDots); int count = 0; // delete previous output line - printf("\r\33[2K [%3.0f%%] [", fraction * 100); + printf("\r [%3.0f%%] [", fraction * 100); for (; count < dotz - 1; count++) putchar('='); putchar('>'); @@ -76,6 +76,7 @@ void ProgressBar::reportProgress(u64_t total, u64_t processed) printBytes(processed); putchar('/'); printBytes(total); + printf("\33[K\r"); if (point == 1000) putchar('\n'); fflush(stdout); From 04b86d1c874a3f27f43220652246369007acb0fa Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sat, 1 Oct 2016 15:22:52 +0300 Subject: [PATCH 163/166] :D:D:D fix #106 --- libgrive/src/util/ProgressBar.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libgrive/src/util/ProgressBar.cc b/libgrive/src/util/ProgressBar.cc index efd71bcf..6fe14443 100644 --- a/libgrive/src/util/ProgressBar.cc +++ b/libgrive/src/util/ProgressBar.cc @@ -53,8 +53,8 @@ void ProgressBar::reportProgress(u64_t total, u64_t processed) // only print progress after >= 0.1% change this->last = point; - // 10 for prefix of percent and 23 for suffix of file size - int availableSize = determineTerminalSize() - 33; + // 10 for prefix of percent and 26 for suffix of file size + int availableSize = determineTerminalSize() - 36; int totalDots; if (availableSize > 100) totalDots = 100; From 7bbb01c3014cc996ec24a7e5c0684f24b604b735 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Tue, 4 Oct 2016 12:25:24 +0300 Subject: [PATCH 164/166] Fix #108 :)) --- libgrive/src/util/ProgressBar.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libgrive/src/util/ProgressBar.cc b/libgrive/src/util/ProgressBar.cc index 6fe14443..6a26b6e9 100644 --- a/libgrive/src/util/ProgressBar.cc +++ b/libgrive/src/util/ProgressBar.cc @@ -47,10 +47,10 @@ void ProgressBar::reportProgress(u64_t total, u64_t processed) processed = total; double fraction = (double)processed/total; - int point = round(fraction*1000); + int point = fraction*1000; if (this->last < 1000 || point != this->last) { - // only print progress after >= 0.1% change + // do not print 100% progress multiple times (it will duplicate the progressbar) this->last = point; // 10 for prefix of percent and 26 for suffix of file size From 6cba67985a9487b8ac77d919ad4666dc87114994 Mon Sep 17 00:00:00 2001 From: Vitaliy Filippov Date: Sun, 15 Oct 2017 14:34:35 +0300 Subject: [PATCH 165/166] Maintained version link --- README.md | 140 +----------------------------------------------------- 1 file changed, 1 insertion(+), 139 deletions(-) diff --git a/README.md b/README.md index 2e2ee468..028e01dd 100644 --- a/README.md +++ b/README.md @@ -1,139 +1 @@ -# Grive2 0.5.1-dev - -28 Sep 2016, Vitaliy Filippov - -http://yourcmc.ru/wiki/Grive2 - -This is the fork of original "Grive" (https://github.com/Grive/grive) Google Drive client -with the support for the new Drive REST API and partial sync. - -Grive can be considered still beta or pre-beta quality. It simply downloads all the files in your -Google Drive into the current directory. After you make some changes to the local files, run -grive again and it will upload your changes back to your Google Drive. New files created locally -or in Google Drive will be uploaded or downloaded respectively. Deleted files will also be "removed". -Currently Grive will NOT destroy any of your files: it will only move the files to a -directory named .trash or put them in the Google Drive trash. You can always recover them. - -There are a few things that Grive does not do at the moment: -- continously wait for changes in file system or in Google Drive to occur and upload. - A sync is only performed when you run Grive. -- symbolic links support. -- support for Google documents. - -These may be added in the future, possibly the next release. - -When Grive is run for the first time, you should use the "-a" argument to grant -permission to Grive to access to your Google Drive. A URL should be printed. -Go to the link. You will need to login to your Google account if you haven't -done so. After granting the permission to Grive, the browser will show you -an authenication code. Copy-and-paste that to the standard input of Grive. - -If everything works fine, Grive will create .grive and .grive_state files in your -current directory. It will also start downloading files from your Google Drive to -your current directory. - -Enjoy! - -## Installation - -For the detailed instructions, see http://yourcmc.ru/wiki/Grive2#Installation - -### Install dependencies - -You need the following libraries: - -- yajl 2.x -- libcurl -- libstdc++ -- libgcrypt -- Boost (Boost filesystem, program_options, regex, unit_test_framework and system are required) -- expat - -There are also some optional dependencies: -- CppUnit (for unit tests) -- libbfd (for backtrace) -- binutils (for libiberty, required for compilation in OpenSUSE, Ubuntu, Arch and etc) - -On a Debian/Ubuntu/Linux Mint machine just run the following command to install all -these packages: - - sudo apt-get install git cmake build-essential libgcrypt11-dev libyajl-dev \ - libboost-all-dev libcurl4-openssl-dev libexpat1-dev libcppunit-dev binutils-dev - -FreeBSD: - - pkg install git cmake boost-libs yajl libgcrypt pkgconf cppunit libbfd - -### Build Debian packages - -On a Debian/Ubuntu/Linux Mint you can use `dpkg-buildpackage` utility from `dpkg-dev` package -to build grive. Just clone the repository, `cd` into it and run - - dpkg-buildpackage -j4 - -### Manual build - -Grive uses cmake to build. Basic install sequence is - - mkdir build - cd build - cmake .. - make -j4 - sudo make install - -## Version History - -### Grive2 v0.5.1-dev - -- no-remote-new and upload-only modes -- ignore regexp does not persist anymore (note that Grive will still track it to not - accidentally delete remote files when changing ignore regexp) -- added options to limit upload and download speed -- faster upload of new and changed files. now Grive uploads files without first calculating - md5 checksum when file is created locally or when its size changes. -- added -P/--progress-bar option to print ASCII progress bar for each processed file (pull request by @svartkanin) - -### Grive2 v0.5 - -- Much faster and more correct synchronisation using local modification time and checksum cache (similar to git index) -- Automatic move/rename detection, -m option removed -- force option works again -- Instead of crashing on sync exceptions Grive will give a warning and attempt to sync failed files again during the next run. -- Revision support works again. Grive 0.4.x always created new revisions for all files during sync, regardless of the absence of the --new-rev option. -- Shared files now sync correctly - -### Grive2 v0.4.2 - -- Option to exclude files by perl regexp -- Reimplemented HTTP response logging for debug purposes -- Use multipart uploads (update metadata and contents at the same time) for improved perfomance & stability -- Bug fixes -- Simple option to move/rename files and directories, via `grive -m oldpath newpath` (by Dylan Wulf, wulfd1@tcnj.edu) - -Known issues: -- force option does not work as documented #51 - -### Grive2 v0.4.1 - -- Bug fixes - -### Grive2 v0.4.0 - -First fork release, by Vitaliy Filippov / vitalif at mail*ru -- Support for the new Google Drive REST API (old "Document List" API is shut down by Google 20 April 2015) -- REAL support for partial sync: syncs only one subdirectory with `grive -s subdir` -- Major refactoring - a lot of dead code removed, JSON-C is not used anymore, API-specific code is split from non-API-specific -- Some stability fixes from Visa Putkinen https://github.com/visap/grive/commits/visa -- Slightly reduce number of syscalls when reading local files. - -### Grive v0.3 - -Bug fix & minor feature release. Fixed bugs: -- #93: missing reference count increment in one of the Json constructors -- #82: retry for HTTP error 500 & 503 -- #77: Fixed a bug where grive crashed on the first run. - -New features: -- #87: support for revisions -- #86: ~~partial sync (contributed by justin at tierramedia.com)~~ that's not partial sync, - that's only support for specifying local path on command line +Maintained version still here: https://github.com/vitalif/grive2 From 7b4797a406f10de1e2549ccf36bbed2e40c414a0 Mon Sep 17 00:00:00 2001 From: Henrique Cesar Ulbrich Date: Wed, 12 Sep 2018 20:01:55 -0300 Subject: [PATCH 166/166] Update README.md --- README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 028e01dd..f71efa1c 100644 --- a/README.md +++ b/README.md @@ -1 +1,8 @@ -Maintained version still here: https://github.com/vitalif/grive2 +GRIVE NO LONGER MAINTAINED +PLEASE USE GRIVE2 + +Grive is DEPRECATED. Please refer to the newest fork Grive2. + +Grive2, WHICH IS ACTIVELY MAINTAINED, is here: https://github.com/vitalif/grive2 + +