APPLY FAILED.

[WMP]

mysql-disk:
  Logical volume "mysql-disk-snapshot" created
  50GB 0:15:28 [55.1MB/s] [                                                                              <=>                                                              ]
51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 928.54 s, 57.8 MB/s

0+1666160 records in
0+1666160 records out
53687091200 bytes (54 GB) copied, 933.374 s, 57.5 MB/s
lvmsync...
Data source: /dev/mapper/virtual-mysql--disk
Sending chunk 0..4095...
Seeking to 0 in /dev/mapper/virtual-mysql--disk
Sending chunk 25904021504..25904025599...
Seeking to 25904021504 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964417024..25964421119...
Seeking to 25964417024 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964421120..25964425215...
Seeking to 25964421120 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964425216..25964429311...
Seeking to 25964425216 in /dev/mapper/virtual-mysql--disk
Sending chunk 20087386112..20087390207...
Seeking to 20087386112 in /dev/mapper/virtual-mysql--disk
Sending chunk 19328221184..19328225279...
Seeking to 19328221184 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964429312..25964433407...
Seeking to 25964429312 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964433408..25964437503...
Seeking to 25964433408 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964437504..25964441599...
Seeking to 25964437504 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964441600..25964445695...
Seeking to 25964441600 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964445696..25964449791...
Seeking to 25964445696 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964449792..25964453887...
Seeking to 25964449792 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964453888..25964457983...
Seeking to 25964453888 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964457984..25964462079...
Seeking to 25964457984 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964462080..25964466175...
Seeking to 25964462080 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964466176..25964470271...
Seeking to 25964466176 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964470272..25964474367...
Seeking to 25964470272 in /dev/mapper/virtual-mysql--disk
Sending chunk 20087390208..20087394303...
Seeking to 20087390208 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964474368..25964478463...
Seeking to 25964474368 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964478464..25964482559...
Seeking to 25964478464 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964482560..25964486655...
Seeking to 25964482560 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964486656..25964490751...
Seeking to 25964486656 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964490752..25964494847...
Seeking to 25964490752 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964494848..25964498943...
Seeking to 25964494848 in /dev/mapper/virtual-mysql--disk
Sending chunk 8192..12287...
Seeking to 8192 in /dev/mapper/virtual-mysql--disk
Sending chunk 19327373312..19327377407...
Seeking to 19327373312 in /dev/mapper/virtual-mysql--disk
Sending chunk 19461591040..19461595135...
Seeking to 19461591040 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964498944..25964503039...
Seeking to 25964498944 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964503040..25964507135...
Seeking to 25964503040 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964507136..25964511231...
Seeking to 25964507136 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964511232..25964515327...
Seeking to 25964511232 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964515328..25964519423...
Seeking to 25964515328 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964519424..25964523519...
Seeking to 25964519424 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964523520..25964527615...
Seeking to 25964523520 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964527616..25964531711...
Seeking to 25964527616 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964531712..25964535807...
Seeking to 25964531712 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964535808..25964539903...
Seeking to 25964535808 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964539904..25964543999...
Seeking to 25964539904 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964544000..25964548095...
Seeking to 25964544000 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964548096..25964552191...
Seeking to 25964548096 in /dev/mapper/virtual-mysql--disk
Sending chunk 19328204800..19328208895...
Seeking to 19328204800 in /dev/mapper/virtual-mysql--disk
Sending chunk 20038578176..20038582271...
Seeking to 20038578176 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964552192..25964556287...
Seeking to 25964552192 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964556288..25964560383...
Seeking to 25964556288 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964560384..25964564479...
Seeking to 25964560384 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964564480..25964568575...
Seeking to 25964564480 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964568576..25964572671...
Seeking to 25964568576 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964572672..25964576767...
Seeking to 25964572672 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964576768..25964580863...
Seeking to 25964576768 in /dev/mapper/virtual-mysql--disk
Sending chunk 50 of 132, 0.20MB/sSending chunk 25964580864..25964584959...
Seeking to 25964580864 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964584960..25964589055...
Seeking to 25964584960 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964589056..25964593151...
Seeking to 25964589056 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964593152..25964597247...
Seeking to 25964593152 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964597248..25964601343...
Seeking to 25964597248 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964601344..25964605439...
Seeking to 25964601344 in /dev/mapper/virtual-mysql--disk
Seeking to 25964601344 in /dev/mapper/virtual-mysql--disk
Sending chunk 20087394304..20087398399...
Seeking to 20087394304 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964605440..25964609535...
Seeking to 25964605440 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964609536..25964613631...
Seeking to 25964609536 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964613632..25964617727...
Seeking to 25964613632 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964617728..25964621823...
Seeking to 25964617728 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964621824..25964625919...
Seeking to 25964621824 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964625920..25964630015...
Seeking to 25964625920 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964630016..25964634111...
Seeking to 25964630016 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964634112..25964638207...
Seeking to 25964634112 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964638208..25964642303...
Seeking to 25964638208 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964642304..25964646399...
Seeking to 25964642304 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964646400..25964650495...
Seeking to 25964646400 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964650496..25964654591...
Seeking to 25964650496 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964654592..25964658687...
Seeking to 25964654592 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964658688..25964662783...
Seeking to 25964658688 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964662784..25964666879...
Seeking to 25964662784 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964666880..25964670975...
Seeking to 25964666880 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964670976..25964675071...
Seeking to 25964670976 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964675072..25964679167...
Seeking to 25964675072 in /dev/mapper/virtual-mysql--disk
Seeking to 25964679168 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964683264..25964687359...
Seeking to 25964683264 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964687360..25964691455...
Seeking to 25964687360 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964691456..25964695551...
Seeking to 25964691456 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964695552..25964699647...
Seeking to 25964695552 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964699648..25964703743...
Seeking to 25964699648 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964703744..25964707839...
Seeking to 25964703744 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964707840..25964711935...
Seeking to 25964707840 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964711936..25964716031...
Seeking to 25964711936 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964716032..25964720127...
Seeking to 25964716032 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964720128..25964724223...
Seeking to 25964720128 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964724224..25964728319...
Seeking to 25964724224 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964728320..25964732415...
Seeking to 25964728320 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964732416..25964736511...
Seeking to 25964732416 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964736512..25964740607...
Seeking to 25964736512 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964740608..25964744703...
Seeking to 25964740608 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964744704..25964748799...
Seeking to 25964744704 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964748800..25964752895...
Seeking to 25964748800 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964752896..25964756991...
Seeking to 25964752896 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964756992..25964761087...
Seeking to 25964756992 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964761088..25964765183...
Seeking to 25964761088 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964765184..25964769279...
Seeking to 25964765184 in /dev/mapper/virtual-mysql--disk
Sending chunk 20087398400..20087402495...
Seeking to 20087398400 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964769280..25964773375...
Seeking to 25964769280 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964773376..25964777471...
Seeking to 25964773376 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964777472..25964781567...
Seeking to 25964777472 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964781568..25964785663...
Seeking to 25964781568 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964785664..25964789759...
Seeking to 25964785664 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964789760..25964793855...
Seeking to 25964789760 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964793856..25964797951...
Seeking to 25964793856 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964797952..25964802047...
Seeking to 25964797952 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964802048..25964806143...
Seeking to 25964802048 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964806144..25964810239...
Seeking to 25964806144 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964810240..25964814335...
Seeking to 25964810240 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964814336..25964818431...
Seeking to 25964814336 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964818432..25964822527...
Seeking to 25964818432 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964822528..25964826623...
Seeking to 25964822528 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964826624..25964830719...
Seeking to 25964826624 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964830720..25964834815...
Seeking to 25964830720 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964834816..25964838911...
Seeking to 25964834816 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964838912..25964843007...
Seeking to 25964838912 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964843008..25964847103...
Seeking to 25964843008 in /dev/mapper/virtual-mysql--disk
Sending chunk 30064836608..30064840703...
Seeking to 30064836608 in /dev/mapper/virtual-mysql--disk
Sending chunk 30064902144..30064906239...
Seeking to 30064902144 in /dev/mapper/virtual-mysql--disk
Sending chunk 30098456576..30098460671...
Seeking to 30098456576 in /dev/mapper/virtual-mysql--disk
Sending chunk 19327901696..19327905791...
Seeking to 19327901696 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964847104..25964851199...
Seeking to 25964847104 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964851200..25964855295...
Seeking to 25964851200 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964855296..25964859391...
Seeking to 25964855296 in /dev/mapper/virtual-mysql--disk
Sending chunk 20039352320..20039356415...
Seeking to 20039352320 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964859392..25964863487...
Seeking to 25964859392 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964863488..25964867583...
Seeking to 25964863488 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964867584..25964871679...
Seeking to 25964867584 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964871680..25964875775...
Seeking to 25964871680 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964875776..25964879871...
Seeking to 25964875776 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964879872..25964883967...
Seeking to 25964879872 in /dev/mapper/virtual-mysql--disk
Sending chunk 25964883968..25964888063...
Seeking to 25964883968 in /dev/mapper/virtual-mysql--disk
Transferred 540672 bytes in 1.01 seconds
You transferred your changes 99296.97x faster than a full dd!
APPLY FAILED.
lvmsync done
cleanup
  Logical volume "mysql-disk-snapshot" successfully removed

[mpalmer]

I'm not sure what's going on here -- my initial impression is that this output appears to be missing the remote-side output. Assuming you're using the latest released version of lvmsync (3.2.0) on the sender side, all output from the remote should be displayed prefixed by remote:, and there shouldn't be a case where the apply fails without producing some output -- even if it's just saying "lvmsync not found", "SSH: Permission denied (publickey)", or some other system-level error message.

So, yeah, mystery. It may be that you've found a corner-case I hadn't considered, that's failing the receiver silently, or perhaps gnomes are eating your errors.

A couple of things to try, to rule out fundamental problems:

1. Run lvmsync --version locally, and make sure that's showing 3.2.0.
2. Run ssh <remote> lvmsync --version and make sure that's showing 3.2.0.

That makes sure that lvmsync is OK on both ends, and can talk to each other. If that's all OK, I'll probably have to make a few lvmsync release with more debugging output on apply, to see why it's failing.

[WMP]

root@monica ~ # lvmsync --version
lvmsync 3.2.0
root@monica ~ # ssh backup lvmsync --version
lvmsync 3.2.0
root@monica ~ #

Maybe is problem with path? lvmsync execute remote commands from bash?

[mpalmer]

On the receiver side, lvmsync does very little -- it opens the destination file and writes chunks where its told. There's not a lot else that it does.

I'll work on releasing a new version of lvmsync with improved debugging capabilities, which should show up what's going on. Look for it by the end of the weekend.

[mpalmer]

OK, version 3.3.0 is out, and now supports --debug. If you could upgrade both sides, then do a run with that option enabled, you should see a whole lot of lines starting with remote: talking about where data blocks are being written to. Hopefully that should shed some light on where things are dying.

[WMP]

root@monica ~ # lvmsync --version
lvmsync 3.3.0
root@monica ~ # ssh backup lvmsync --version
lvmsync 3.3.0
root@monica ~ # bash -x lvmbackup.sh -s backup -v vg0 -o virtual mysql-disk
+ server=
+ virtualgroup_server=
+ virtualgroup_original=
+ getopts “s:v:o:h” option
+ case $option in
+ server=backup
+ getopts “s:v:o:h” option
+ case $option in
+ virtualgroup_server=vg0
+ getopts “s:v:o:h” option
+ case $option in
+ virtualgroup_original=virtual
+ getopts “s:v:o:h” option
+ shift 6
+ [[ -z backup ]]
+ [[ -z vg0 ]]  
+ [[ -z virtual ]]
+ [[ 1 -lt 1 ]] 
+ ((  1  ))
+ echo mysql-disk:
mysql-disk:
+ [[ ! -e /dev/virtual/mysql-disk ]]
++ ssh backup 'test -e /dev/vg0/mysql-disk && echo 1 || echo 0'
+ FileExists=1  
++ lvdisplay --units m virtual/mysql-disk
++ awk '{print $3}'
++ sed -e 's/\..*//'
++ grep 'LV Size'
+ size=51200
+ snapshot_size=5120
+ lvcreate --snapshot -L 5120m -n mysql-disk-snapshot virtual/mysql-disk
  Logical volume "mysql-disk-snapshot" created
+ [[ 1 -eq 0 ]] 
+ echo lvmsync...
lvmsync...
++ date +%Y-%m-%d-%H:%M
+ lvmsync --debug /dev/virtual/mysql-disk-snapshot backup:/dev/vg0/mysql-disk --snapback /snapback/mysql-disk/2014-11-07-11:44
Data source: /dev/mapper/virtual-mysql--disk
Sending chunk 0..4095...
Sending chunk 590241792..590245887...
Sending chunk 25904021504..25904025599...
Sending chunk 25908326400..25908330495...
Sending chunk 25908330496..25908334591...
Sending chunk 25908334592..25908338687...
Sending chunk 563085312..563089407...
Sending chunk 563089408..563093503...
Sending chunk 563093504..563097599...
Sending chunk 563097600..563101695...
Sending chunk 563101696..563105791...
Sending chunk 563105792..563109887...
Sending chunk 563109888..563113983...
Sending chunk 563113984..563118079...
Sending chunk 563118080..563122175...
Sending chunk 563122176..563126271...
Sending chunk 563126272..563130367...
Sending chunk 563130368..563134463...
Sending chunk 563134464..563138559...
Sending chunk 563138560..563142655...
Sending chunk 563142656..563146751...
Sending chunk 563146752..563150847...
Sending chunk 563150848..563154943...
Sending chunk 563154944..563159039...
Sending chunk 563159040..563163135...
Sending chunk 563163136..563167231...
Sending chunk 563167232..563171327...
Sending chunk 563171328..563175423...
Sending chunk 563175424..563179519...
Sending chunk 563179520..563183615...
Sending chunk 563183616..563187711...
Sending chunk 563187712..563191807...
Sending chunk 563191808..563195903...
Sending chunk 563195904..563199999...
Sending chunk 563200000..563204095...
Sending chunk 563204096..563208191...
Sending chunk 563208192..563212287...
Sending chunk 563212288..563216383...
Sending chunk 563216384..563220479...
Sending chunk 563220480..563224575...
Sending chunk 563224576..563228671...
Sending chunk 563228672..563232767...
Sending chunk 563232768..563236863...
Sending chunk 563236864..563240959...
Sending chunk 563240960..563245055...
Sending chunk 563245056..563249151...
Sending chunk 563249152..563253247...
Sending chunk 563253248..563257343...
Sending chunk 563257344..563261439...
Sending chunk 563261440..563265535...
Sending chunk 50 of 115, 0.13MB/sSending chunk 563265536..563269631...
Sending chunk 563269632..563273727...
Sending chunk 563273728..563277823...
Sending chunk 563277824..563281919...
Sending chunk 563281920..563286015...
Sending chunk 563286016..563290111...
Sending chunk 563290112..563294207...
Sending chunk 563294208..563298303...
Sending chunk 563298304..563302399...
Sending chunk 563302400..563306495...
Sending chunk 563306496..563310591...
Sending chunk 563310592..563314687...
Sending chunk 563314688..563318783...
Sending chunk 563318784..563322879...
Sending chunk 563322880..563326975...
Sending chunk 563326976..563331071...
Sending chunk 563331072..563335167...
Sending chunk 563335168..563339263...
Sending chunk 563339264..563343359...
Sending chunk 563343360..563347455...
Sending chunk 563347456..563351551...
Sending chunk 563351552..563355647...
Sending chunk 563355648..563359743...
Sending chunk 563359744..563363839...
Sending chunk 563363840..563367935...
Sending chunk 563367936..563372031...
Sending chunk 563372032..563376127...
Sending chunk 563376128..563380223...
Sending chunk 563380224..563384319...
Sending chunk 563384320..563388415...
Sending chunk 563388416..563392511...
Sending chunk 563392512..563396607...
Sending chunk 563396608..563400703...
Sending chunk 563400704..563404799...
Sending chunk 563404800..563408895...
Sending chunk 563408896..563412991...
Sending chunk 563412992..563417087...
Sending chunk 563417088..563421183...
Sending chunk 563421184..563425279...
Sending chunk 563425280..563429375...
Sending chunk 563429376..563433471...
Sending chunk 563433472..563437567...
Sending chunk 563437568..563441663...
Sending chunk 563441664..563445759...
Sending chunk 563445760..563449855...
Sending chunk 563449856..563453951...
Sending chunk 563453952..563458047...
Sending chunk 563458048..563462143...
Sending chunk 563462144..563466239...
Sending chunk 563466240..563470335...
Sending chunk 563470336..563474431...
Sending chunk 563474432..563478527...
Sending chunk 25908338688..25908342783...
Sending chunk 25908342784..25908346879...
Sending chunk 25908346880..25908350975...
Sending chunk 25908350976..25908355071...
Sending chunk 25908355072..25908359167...
Sending chunk 25908359168..25908363263...
Sending chunk 25908363264..25908367359...
Sending chunk 25908367360..25908371455...
Sending chunk 25908371456..25908375551...
Sending chunk 568803328..568807423...
Sending chunk 568807424..568811519...
Sending chunk 568811520..568815615...
Sending chunk 568815616..568819711...
Transferred 471040 bytes in 1.54 seconds
You transferred your changes 113975.65x faster than a full dd!
FATAL ERROR: APPLY FAILED.
+ echo 'lvmsync done'
lvmsync done
+ echo cleanup  
cleanup
+ lvremove -f /dev/virtual/mysql-disk-snapshot
  Logical volume "mysql-disk-snapshot" successfully removed
+ shift
+ ((  0  ))
root@monica ~ #

[WMP]

Maybe this is problem?

root@monica ~ # ssh backup which lvmsync
/usr/local/bin/lvmsync
root@monica ~ # ssh backup cat `which lvmsync`
#!/usr/bin/env bash

# load rvm ruby
source /usr/local/rvm/environments/ruby-2.0.0-p576

ruby /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync $@
root@monica ~ #

[WMP]

Ok, i thinks i found error:

root@backup ~ # cat /usr/local/bin/lvmsync 
#!/usr/bin/env bash

# load rvm ruby
source /usr/local/rvm/environments/ruby-2.0.0-p576

echo $@ >> /var/log/lvmsync.log
eval 'ruby /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync $@' 2>&1 | tee -a "/var/log/lvmsync.log"
root@backup ~ # 

root@monica ~ # bash lvmbackup.sh -s backup -v vg0 -o virtual mysql-disk
mysql-disk:
  Logical volume "mysql-disk-snapshot" created
lvmsync...
Data source: /dev/mapper/virtual-mysql--disk
Sending chunk 0..4095...
Transferred 4096 bytes in 0.04 seconds
You transferred your changes 13107200.00x faster than a full dd!
lvmsync done
cleanup
  Logical volume "mysql-disk-snapshot" successfully removed
root@monica ~ # 

root@backup ~ # cat /var/log/lvmsync.log 
--apply - --snapback /snapback/mysql-disk/2014-11-07-12:06 -v -d /dev/vg0/mysql-disk
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:129:in `initialize': No such file or directory - /snapback/mysql-disk/2014-11-07-12:06 (Errno::ENOENT)
        from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:129:in `open'
        from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:129:in `run_apply'
        from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:96:in `main'
        from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:357:in `<top (required)>'
        from /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync:23:in `load'
        from /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync:23:in `<main>'
--apply - --snapback /snapback/mysql-disk/2014-11-07-12:07 -v -d /dev/vg0/mysql-disk
Writing changed data to "/dev/vg0/mysql-disk"
Seeking to 0
Wrote 4096 bytes at 0
--apply - --snapback /snapback/mysql-disk/2014-11-07-12:08 -v -d /dev/vg0/mysql-disk
Writing changed data to "/dev/vg0/mysql-disk"
Seeking to 0
Wrote 4096 bytes at 0
root@backup ~ # 

You must fix --verbose and --debug ;)

[Jasperswaagman]

If anyone else stumbles into this. Make sure the directory you want to store the snapback in exists on the server/remote. If it does not exist it will give a:
FATAL ERROR: APPLY FAILED.