From 9fab6a2203e98cf6666d8d9978298666586b1df4 Mon Sep 17 00:00:00 2001 From: alawvt Date: Tue, 25 May 2021 14:46:01 -0400 Subject: [PATCH 01/23] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 218e2cb1a7..65fa18d8ca 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,6 @@ -# DSpace +# DSpace customized for [VTechWorks](https://vtechworks.lib.vt.edu/) +hosted by the University Libraries at Virginia Tech [![Build Status](https://travis-ci.org/DSpace/DSpace.png?branch=master)](https://travis-ci.org/DSpace/DSpace) From e89d62b9939358ca35034eb455afc619c5ac25ce Mon Sep 17 00:00:00 2001 From: alawvt Date: Tue, 25 May 2021 14:49:20 -0400 Subject: [PATCH 02/23] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 65fa18d8ca..8c62836fd8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DSpace customized for [VTechWorks](https://vtechworks.lib.vt.edu/) -hosted by the University Libraries at Virginia Tech +University Libraries, Virginia Tech [![Build Status](https://travis-ci.org/DSpace/DSpace.png?branch=master)](https://travis-ci.org/DSpace/DSpace) From bcaa304876732498b93ebdb284b1fc5f655de3e4 Mon Sep 17 00:00:00 2001 From: alawvt Date: Tue, 25 May 2021 14:53:24 -0400 Subject: [PATCH 03/23] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8c62836fd8..e56029cc6a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DSpace customized for [VTechWorks](https://vtechworks.lib.vt.edu/) -University Libraries, Virginia Tech +[University Libraries](https://lib.vt.edu/), [Virginia Tech](https://vt.edu/) [![Build Status](https://travis-ci.org/DSpace/DSpace.png?branch=master)](https://travis-ci.org/DSpace/DSpace) From 0c5950f5908057238afc070bc420d5f61d85c17d Mon Sep 17 00:00:00 2001 From: alawvt Date: Tue, 1 Jun 2021 15:37:09 -0400 Subject: [PATCH 04/23] Update news-xmlui.xml with new link to VTechData and stats sheet --- dspace/config/news-xmlui.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dspace/config/news-xmlui.xml b/dspace/config/news-xmlui.xml index a5985fb06d..540707cf81 100644 --- a/dspace/config/news-xmlui.xml +++ b/dspace/config/news-xmlui.xml @@ -5,11 +5,11 @@ VTechWorks

VTechWorks provides global access to Virginia Tech scholarship. VTechWorks includes journal articles, books, theses, dissertations, conference papers, slide presentations, technical reports, working papers, administrative documents, videos, images, and more by faculty, students, and staff. Write vtechworks@vt.edu to get help adding your content to VTechWorks. Read an article about VTechWorks or visit the Open@VT blog to learn about current VTechWorks activities.

-

Want to publish a standalone dataset? Visit VTechData.

+

Want to publish a standalone dataset? Visit Virginia Tech Data Repository.

Faculty can deposit items to VTechWorks from Elements (EFARs). Visit the Provost's Elements page to learn more and to log in to Elements.

-

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

+

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

From 04a8dd4c0b2c0443fd5dbc528dcd59b3284dac23 Mon Sep 17 00:00:00 2001 From: Paul Mather Date: Fri, 11 Jun 2021 15:33:36 -0400 Subject: [PATCH 05/23] Limit the number of log files that are retained. The current log settings configure the logger with "MaxLogs=0". This means no old log files are deleted after log rotation. Instead, we avoid runaway log file accumulation by setting a non-zero value for "MaxLogs". --- dspace/config/log4j-solr.properties | 2 ++ dspace/config/log4j.properties | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/dspace/config/log4j-solr.properties b/dspace/config/log4j-solr.properties index c94e384d81..41d07b49f2 100644 --- a/dspace/config/log4j-solr.properties +++ b/dspace/config/log4j-solr.properties @@ -31,6 +31,8 @@ log4j.appender.CONSOLE.layout.ConversionPattern=%-4r [%t] %-5p %c %x \u2013 %m%n log4j.appender.file=org.apache.log4j.DailyRollingFileAppender # Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files log4j.appender.file.DatePattern='.'yyyy-MM-dd +# The number of log files to keep, or 0 to keep them all +log4j.appender.file.MaxLogs=60 #- File to log to and log format log4j.appender.file.File=${log.dir}/solr.log diff --git a/dspace/config/log4j.properties b/dspace/config/log4j.properties index 238d0d8fbf..02231b3dbf 100644 --- a/dspace/config/log4j.properties +++ b/dspace/config/log4j.properties @@ -49,7 +49,7 @@ log4j.appender.A1.File=${log.dir}/dspace.log # Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files log4j.appender.A1.DatePattern=yyyy-MM-dd # The number of log files to keep, or 0 to keep them all -log4j.appender.A1.MaxLogs=0 +log4j.appender.A1.MaxLogs=60 # A1 uses PatternLayout. log4j.appender.A1.layout=org.apache.log4j.PatternLayout log4j.appender.A1.layout.ConversionPattern=%d %-5p %c @ %m%n @@ -70,7 +70,7 @@ log4j.appender.A2.File=${log.dir}/checker.log # Set this to yyyy-MM-DD for daily log files, or yyyy-MM for monthly files log4j.appender.A2.DatePattern=yyyy-MM-dd # The number of log files to keep, or 0 to keep them all -log4j.appender.A2.MaxLogs=0 +log4j.appender.A2.MaxLogs=60 # A2 uses PatternLayout. log4j.appender.A2.layout=org.apache.log4j.PatternLayout log4j.appender.A2.layout.ConversionPattern=%m%n From 5a254d786bba90c2aa4f63c8770ba2ad2997e258 Mon Sep 17 00:00:00 2001 From: alawvt Date: Tue, 15 Jun 2021 13:35:29 -0400 Subject: [PATCH 06/23] Update news-xmlui.xml Change stats link to FY2021-2022 sheet. --- dspace/config/news-xmlui.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dspace/config/news-xmlui.xml b/dspace/config/news-xmlui.xml index 540707cf81..12b4f1ee95 100644 --- a/dspace/config/news-xmlui.xml +++ b/dspace/config/news-xmlui.xml @@ -9,7 +9,7 @@

Faculty can deposit items to VTechWorks from Elements (EFARs). Visit the Provost's Elements page to learn more and to log in to Elements.

-

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

+

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

From 60ba5c4de7c5044d8added67c6b84456fa02f59f Mon Sep 17 00:00:00 2001 From: alawvt Date: Tue, 27 Jul 2021 15:13:07 -0400 Subject: [PATCH 07/23] Ds 3914 (#756) * add files for community-filiator fix DS-3914 * name CommunityFiliator.jave correctly * remove incorrectly name CommunityFilator.java --- .../dspace/administer/CommunityFiliator.java | 292 ++++++++++++++++++ .../java/org/dspace/content/Community.java | 274 ++++++++++++++++ 2 files changed, 566 insertions(+) create mode 100644 dspace/modules/additions/src/main/java/org/dspace/administer/CommunityFiliator.java create mode 100644 dspace/modules/additions/src/main/java/org/dspace/content/Community.java diff --git a/dspace/modules/additions/src/main/java/org/dspace/administer/CommunityFiliator.java b/dspace/modules/additions/src/main/java/org/dspace/administer/CommunityFiliator.java new file mode 100644 index 0000000000..03bdccdb70 --- /dev/null +++ b/dspace/modules/additions/src/main/java/org/dspace/administer/CommunityFiliator.java @@ -0,0 +1,292 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.administer; + +import java.io.IOException; +import java.sql.SQLException; +import java.util.List; +import java.util.UUID; + +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.CommandLineParser; +import org.apache.commons.cli.HelpFormatter; +import org.apache.commons.cli.Options; +import org.apache.commons.cli.PosixParser; +import org.apache.commons.collections.CollectionUtils; +import org.dspace.authorize.AuthorizeException; +import org.dspace.content.Community; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.CommunityService; +import org.dspace.core.Constants; +import org.dspace.core.Context; +import org.dspace.handle.factory.HandleServiceFactory; +import org.dspace.handle.service.HandleService; + +/** + * A command-line tool for setting/removing community/sub-community + * relationships. Takes community DB Id or handle arguments as inputs. + * + * @author rrodgers + * @version $Revision$ + */ + +public class CommunityFiliator +{ + + protected CommunityService communityService; + protected HandleService handleService; + + public CommunityFiliator() { + communityService = ContentServiceFactory.getInstance().getCommunityService(); + handleService = HandleServiceFactory.getInstance().getHandleService(); + } + + /** + * + * @param argv arguments + * @throws Exception if error + */ + public static void main(String[] argv) throws Exception + { + // create an options object and populate it + CommandLineParser parser = new PosixParser(); + + Options options = new Options(); + + options.addOption("s", "set", false, "set a parent/child relationship"); + options.addOption("r", "remove", false, + "remove a parent/child relationship"); + options.addOption("p", "parent", true, + "parent community (handle or database ID)"); + options.addOption("c", "child", true, + "child community (handle or databaseID)"); + options.addOption("h", "help", false, "help"); + + CommandLine line = parser.parse(options, argv); + + String command = null; // set or remove + String parentID = null; + String childID = null; + + if (line.hasOption('h')) + { + HelpFormatter myhelp = new HelpFormatter(); + myhelp.printHelp("CommunityFiliator\n", options); + System.out + .println("\nestablish a relationship: CommunityFiliator -s -p parentID -c childID"); + System.out + .println("remove a relationship: CommunityFiliator -r -p parentID -c childID"); + + System.exit(0); + } + + if (line.hasOption('s')) + { + command = "set"; + } + + if (line.hasOption('r')) + { + command = "remove"; + } + + if (line.hasOption('p')) // parent + { + parentID = line.getOptionValue('p'); + } + + if (line.hasOption('c')) // child + { + childID = line.getOptionValue('c'); + } + + // now validate + // must have a command set + if (command == null) + { + System.out + .println("Error - must run with either set or remove (run with -h flag for details)"); + System.exit(1); + } + + if ("set".equals(command) || "remove".equals(command)) + { + if (parentID == null) + { + System.out.println("Error - a parentID must be specified (run with -h flag for details)"); + System.exit(1); + } + + if (childID == null) + { + System.out.println("Error - a childID must be specified (run with -h flag for details)"); + System.exit(1); + } + } + + CommunityFiliator filiator = new CommunityFiliator(); + Context c = new Context(); + + // we are superuser! + c.turnOffAuthorisationSystem(); + + try + { + // validate and resolve the parent and child IDs into commmunities + Community parent = filiator.resolveCommunity(c, parentID); + Community child = filiator.resolveCommunity(c, childID); + + if (parent == null) + { + System.out.println("Error, parent community cannot be found: " + + parentID); + System.exit(1); + } + + if (child == null) + { + System.out.println("Error, child community cannot be found: " + + childID); + System.exit(1); + } + + if ("set".equals(command)) + { + filiator.filiate(c, parent, child); + } + else + { + filiator.defiliate(c, parent, child); + } + } + catch (SQLException sqlE) + { + System.out.println("Error - SQL exception: " + sqlE.toString()); + } + catch (AuthorizeException authE) + { + System.out.println("Error - Authorize exception: " + + authE.toString()); + } + catch (IOException ioE) + { + System.out.println("Error - IO exception: " + ioE.toString()); + } + } + + /** + * + * @param c context + * @param parent parent Community + * @param child child community + * @throws SQLException if database error + * @throws AuthorizeException if authorize error + * @throws IOException if IO error + */ + public void filiate(Context c, Community parent, Community child) + throws SQLException, AuthorizeException, IOException + { + // check that a valid filiation would be established + // first test - proposed child must currently be an orphan (i.e. + // top-level) + Community childDad = CollectionUtils.isNotEmpty(child.getParentCommunities()) ? child.getParentCommunities().iterator().next() : null; + + if (childDad != null) + { + System.out.println("Error, child community: " + child.getID() + + " already a child of: " + childDad.getID()); + System.exit(1); + } + + // second test - circularity: parent's parents can't include proposed + // child + List parentDads = parent.getParentCommunities(); + if (parentDads.contains(child)) + { + System.out.println( + "Error, circular parentage - child is parent of parent"); + System.exit(1); + } + + // everthing's OK + communityService.addSubcommunity(c, parent, child); + + // complete the pending transaction + c.complete(); + System.out.println("Filiation complete. Community: '" + parent.getID() + + "' is parent of community: '" + child.getID() + "'"); + } + + /** + * + * @param c context + * @param parent parent Community + * @param child child community + * @throws SQLException if database error + * @throws AuthorizeException if authorize error + * @throws IOException if IO error + */ + public void defiliate(Context c, Community parent, Community child) + throws SQLException, AuthorizeException, IOException + { + // verify that child is indeed a child of parent + List parentKids = parent.getSubcommunities(); + if (!parentKids.contains(child)) + { + System.out + .println("Error, child community not a child of parent community"); + System.exit(1); + } + + // OK remove the mappings - but leave the community, which will become + // top-level + child.removeParentCommunity(parent); + parent.removeSubCommunity(child); + communityService.update(c, child); + communityService.update(c, parent); + + // complete the pending transaction + c.complete(); + System.out.println("Defiliation complete. Community: '" + child.getID() + + "' is no longer a child of community: '" + parent.getID() + + "'"); + } + + /** + * Find a community by ID + * @param c context + * @param communityID community ID + * @return Community object + * @throws SQLException if database error + */ + protected Community resolveCommunity(Context c, String communityID) + throws SQLException + { + Community community = null; + + if (communityID.indexOf('/') != -1) + { + // has a / must be a handle + community = (Community) handleService.resolveToObject(c, + communityID); + + // ensure it's a community + if ((community == null) + || (community.getType() != Constants.COMMUNITY)) + { + community = null; + } + } + else + { + community = communityService.find(c, UUID.fromString(communityID)); + } + + return community; + } +} \ No newline at end of file diff --git a/dspace/modules/additions/src/main/java/org/dspace/content/Community.java b/dspace/modules/additions/src/main/java/org/dspace/content/Community.java new file mode 100644 index 0000000000..d8626dd082 --- /dev/null +++ b/dspace/modules/additions/src/main/java/org/dspace/content/Community.java @@ -0,0 +1,274 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.content; + +import org.apache.commons.lang.builder.HashCodeBuilder; +import org.apache.log4j.Logger; +import org.dspace.content.comparator.NameAscendingComparator; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.CommunityService; +import org.dspace.core.*; +import org.dspace.eperson.Group; +import org.hibernate.annotations.CacheConcurrencyStrategy; +import org.hibernate.proxy.HibernateProxyHelper; + +import javax.persistence.*; +import java.util.*; + +/** + * Class representing a community + *

+ * The community's metadata (name, introductory text etc.) is loaded into' + * memory. Changes to this metadata are only reflected in the database after + * update is called. + * + * @author Robert Tansley + * @version $Revision$ + */ +@Entity +@Table(name="community") +@Cacheable +@org.hibernate.annotations.Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE, include = "non-lazy") +public class Community extends DSpaceObject implements DSpaceObjectLegacySupport +{ + /** log4j category */ + private static final Logger log = Logger.getLogger(Community.class); + + @Column(name="community_id", insertable = false, updatable = false) + private Integer legacyId; + + @ManyToMany(fetch = FetchType.LAZY) + @JoinTable( + name = "community2community", + joinColumns = {@JoinColumn(name = "parent_comm_id") }, + inverseJoinColumns = {@JoinColumn(name = "child_comm_id") } + ) + private Set subCommunities = new HashSet<>(); + + @ManyToMany(fetch = FetchType.LAZY, mappedBy = "subCommunities") + private Set parentCommunities = new HashSet<>(); + + @ManyToMany(fetch = FetchType.LAZY, mappedBy = "communities", cascade = {CascadeType.PERSIST}) + private Set collections = new HashSet<>(); + + @OneToOne + @JoinColumn(name = "admin") + /** The default group of administrators */ + private Group admins; + + /** The logo bitstream */ + @OneToOne + @JoinColumn(name = "logo_bitstream_id") + private Bitstream logo = null; + + // Keys for accessing Community metadata + public static final String COPYRIGHT_TEXT = "copyright_text"; + public static final String INTRODUCTORY_TEXT = "introductory_text"; + public static final String SHORT_DESCRIPTION = "short_description"; + public static final String SIDEBAR_TEXT = "side_bar_text"; + + @Transient + protected transient CommunityService communityService; + + /** + * Protected constructor, create object using: + * {@link org.dspace.content.service.CommunityService#create(Community, Context)} + * or + * {@link org.dspace.content.service.CommunityService#create(Community, Context, String)} + * + */ + protected Community() + { + + } + + void addSubCommunity(Community subCommunity) + { + subCommunities.add(subCommunity); + setModified(); + } + + public void removeSubCommunity(Community subCommunity) + { + subCommunities.remove(subCommunity); + setModified(); + } + + /** + * Get the logo for the community. null is return if the + * community does not have a logo. + * + * @return the logo of the community, or null + */ + public Bitstream getLogo() + { + return logo; + } + + void setLogo(Bitstream logo) { + this.logo = logo; + setModified(); + } + + /** + * Get the default group of administrators, if there is one. Note that the + * authorization system may allow others to be administrators for the + * community. + *

+ * The default group of administrators for community 100 is the one called + * community_100_admin. + * + * @return group of administrators, or null if there is no + * default group. + */ + public Group getAdministrators() + { + return admins; + } + + void setAdmins(Group admins) { + this.admins = admins; + setModified(); + } + + /** + * Get the collections in this community. Throws an SQLException because + * creating a community object won't load in all collections. + * + * @return array of Collection objects + */ + public List getCollections() + { + // We return a copy because we do not want people to add elements to this collection directly. + // We return a list to maintain backwards compatibility + Collection[] output = collections.toArray(new Collection[]{}); + Arrays.sort(output, new NameAscendingComparator()); + return Arrays.asList(output); + } + + void addCollection(Collection collection) + { + collections.add(collection); + } + + void removeCollection(Collection collection) + { + collections.remove(collection); + } + + /** + * Get the immediate sub-communities of this community. Throws an + * SQLException because creating a community object won't load in all + * collections. + * + * @return array of Community objects + */ + public List getSubcommunities() + { + // We return a copy because we do not want people to add elements to this collection directly. + // We return a list to maintain backwards compatibility + Community[] output = subCommunities.toArray(new Community[]{}); + Arrays.sort(output, new NameAscendingComparator()); + return Arrays.asList(output); + } + + /** + * Return the parent community of this community, or null if the community + * is top-level + * + * @return the immediate parent community, or null if top-level + */ + public List getParentCommunities() + { + // We return a copy because we do not want people to add elements to this collection directly. + // We return a list to maintain backwards compatibility + Community[] output = parentCommunities.toArray(new Community[]{}); + Arrays.sort(output, new NameAscendingComparator()); + return Arrays.asList(output); + } + + void addParentCommunity(Community parentCommunity) { + parentCommunities.add(parentCommunity); + } + + void clearParentCommunities(){ + parentCommunities.clear(); + } + + public void removeParentCommunity(Community parentCommunity) + { + parentCommunities.remove(parentCommunity); + setModified(); + } + + /** + * Return true if other is the same Community + * as this object, false otherwise + * + * @param other + * object to compare to + * + * @return true if object passed in represents the same + * community as this object + */ + @Override + public boolean equals(Object other) + { + if (other == null) + { + return false; + } + Class objClass = HibernateProxyHelper.getClassWithoutInitializingProxy(other); + if (this.getClass() != objClass) + { + return false; + } + final Community otherCommunity = (Community) other; + if (!this.getID().equals(otherCommunity.getID() )) + { + return false; + } + + return true; + } + + @Override + public int hashCode() + { + return new HashCodeBuilder().append(getID()).toHashCode(); + } + + /** + * return type found in Constants + * @return Community type + */ + @Override + public int getType() + { + return Constants.COMMUNITY; + } + + @Override + public String getName() { + String value = getCommunityService().getMetadataFirstValue(this, MetadataSchema.DC_SCHEMA, "title", null, Item.ANY); + return value == null ? "" : value; + } + + @Override + public Integer getLegacyId() { + return legacyId; + } + + private CommunityService getCommunityService() { + if(communityService == null) + { + communityService = ContentServiceFactory.getInstance().getCommunityService(); + } + return communityService; + } +} \ No newline at end of file From 221411e7393a94d77cf1ad8a61d2e4b740ab6513 Mon Sep 17 00:00:00 2001 From: alawvt Date: Wed, 11 Aug 2021 13:34:05 -0400 Subject: [PATCH 08/23] Advanced search (#760) * Add more advanced search filters and filter messages --- dspace/config/spring/api/discovery.xml | 74 +++++++++++++++++-- .../aspects/Discovery/i18n/messages.xml | 17 +++-- 2 files changed, 77 insertions(+), 14 deletions(-) diff --git a/dspace/config/spring/api/discovery.xml b/dspace/config/spring/api/discovery.xml index 9496744b7d..64456be33e 100644 --- a/dspace/config/spring/api/discovery.xml +++ b/dspace/config/spring/api/discovery.xml @@ -123,10 +123,15 @@ - + - + + + + + + @@ -245,10 +250,15 @@ - + - + + + + + + @@ -337,14 +347,16 @@ - - - - + + + + + + @@ -636,6 +648,15 @@ + + + + + dc.subject.lcc + + + + @@ -663,6 +684,15 @@ + + + + + dc.title.serial + + + + @@ -682,6 +712,25 @@ + + + + + dc.publisher + + + + + + + + + dc.rights + dc.rights.uri + + + + @@ -721,6 +770,15 @@ + + + + + dc.description.version + + + + diff --git a/dspace/modules/xmlui/src/main/resources/aspects/Discovery/i18n/messages.xml b/dspace/modules/xmlui/src/main/resources/aspects/Discovery/i18n/messages.xml index 01f322128c..4460dd852f 100644 --- a/dspace/modules/xmlui/src/main/resources/aspects/Discovery/i18n/messages.xml +++ b/dspace/modules/xmlui/src/main/resources/aspects/Discovery/i18n/messages.xml @@ -51,7 +51,6 @@ Committee Member Thesis Degree Level Department - Series Patent Type Inventor Assignee @@ -87,6 +86,7 @@ Series + Browsing by: Author Browsing by: Title Browsing by: Subject @@ -109,15 +109,21 @@ Browsing by: Department Browsing by: Issue date + Title Date issued Keyword Abstract - Series - Mime-Type - Sponsor + Call Number Identifier + Journal Title Language (ISO) + Publisher + Rights + Sponsor + Series + Version + Mime-Type Coverage Technical Report No. Committee Member @@ -138,6 +144,7 @@ Now showing items {0}-{1} + Filter by: Author Author Author @@ -160,8 +167,6 @@ Filter by: Assignee Filter by: Inventor Filter by: Patent Type - Date Issued - Starts with Or enter first few letters: From 7156c7024ff9d70462ecad42973a00ad8d58106f Mon Sep 17 00:00:00 2001 From: alawvt Date: Wed, 11 Aug 2021 14:20:30 -0400 Subject: [PATCH 09/23] add new VT favicon (#762) --- .../themes/vtmirage2/images/favicon.ico | Bin 22486 -> 1682 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/images/favicon.ico b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/images/favicon.ico index 7274e665188a1c3c53186320f615f12f0695152c..f353a5f98c34c21cd43a64c9fd42fefaa2ca6af9 100644 GIT binary patch literal 1682 zcmeAS@N?(olHy`uVBq!ia0vp^1|ZDA1SD@Hc-mj#PnPRIHZt82`Ti~3Uk?B!Ylp0*+7m{3+ootz+WN)WnQ(*-(AUCxn zQK2F?C$HG5!d3}vt`(3C64qBz04piUwpD^SD#ABF!8yMuRl!uxOgGuk*h0bFQqR!T z(!$6@N5ROz&`jUJQs2--*TB%qz|zXVPyq^*fVLH-q*(>IxIyg#@@$ndN=gc>^!3Zj z%k|2Q_413-^$jg8E%gnI^o@*kfhu&1EAvVcD|GXUm0>2hq!uR^WfqiV=I1GZOiWD5 zFD$Tv3bSNU;+l1ennz|zM-B0$V)JVzP|XC=H|jx7ncO3BHWAB;NpiyW)Z+ZoqGVvir744~DzI`cN=+=uFAB-e&w+(vKt_H^esM;Afr7I$DAddqG<*}2 zGxI=#nqXbNzE+-j#U+V($*G<$wn{*A^fEJ3tPGtDO^jU3EiD~QT@4Lg%`M$5EF2x3 z99_*^&CHxkond-i@{>z*Q}aq-dQ%X39dYUfC5YStpv^9+MVV!(DQ-pixe8#9TV>*Q zi#bm7pn6kqyTt;hUVWfr^g+>!6x}c(U>X83;fWW>fhYgeJYbqH0w(T16)I`KJj3AW z;uunK>&f(h9$ynLhLhJT!h3H3G+0~?({9W)N-)Zb+X%bK8uA%cDy*? z@JNkQ#gB>4ATz>b))YU3gAN5vJ9hkh`uAJKj_oV=Y-2Y3-nM1?_q)H(mH+>L=d;}N zxEQ(jmfmk`&P!%d zD?eu*bKq9t&5IsPRjAo`W&4#KZU0)>oi=xR#dou|)LyudD4jiZXN*SHcY7TVzUbfP zDIBm zoRIEj9l) zNHG_(>-!5UZ0^im8}^Jva7mou~Ua=*k28 z)-gs(sy1BR{)2(tZkt70Y(;X&K~~pgEazoEUknGz>oNcI*!M4La(~#70%?gK7y+S3j3^P6ndYYV~TNWxbY_Y34$CzyEn29%g_USxU3-K@E|;$XuJba5D~R%C4Z+n5gkR`z4w-WyAe$TWcpS#-kzw1pF-b?Qc=#LdlY(n zYlymZCkjBBNFo#~)q@B1Rq@}-!O1C_R;@ZpLI_&C_%NlV714$bCuzlsB6|P*Li*#6 z5_;s3b=0$G8ZBE^NQ)L7rs(Kn$a4&ZX@OWE28af9WN#mdXB^e9zY@>7P}c)>J#bg| zfJzDqi}>(p?CsG+C2tohmAAJig15>?icooYdNuO&^x_!KqOw=nEB~y1%cYl9p>ng6 zW9;1J{4RF(b_n*)uJX4Nz8f^ex4rpWrKzG2`D6r0s_|1$&+X4+i!oQZ@&3~UVb^9MvT}(_urp} z^wj|G!@M6yA}*DN58p!L#%-c==YF7%KR%0clE}p+p4zob1Whz`>5@!Wu3VA!{+l-i z_V?X%?b;1WOgw`3Tx!uGfmA9j5KE048z>@TA0;P$OuznmomQ{jO+Wqg8+clW^T|M+ zI<3X`FUVg)3m5L9OP6j?Qc?jOKVD3eC!45YLoGQuX~EBC`uy{Yl#+6Se*XCyg@^B= z>(@)@yYDVh+qTK{%P-eae*x{=_c^`$ZWg(@>8L>iElrt{Nk@)cK>a6B&sEyL{|lNn zEfamGLmz5s(j*fdJ$eD{V|pFVA1$p}Q-HXy=1=Oo&?_21z06Yz8cZO_9wZJ_q;lkj~R=i*J8 zJv*De{`x%Hn@?U|Q5e4{l((0Pi_g=NCEMxd&71W7_m|P`IMC+Pm@zptZ{B{oaN!dA z@ETpde2rpb3u(@rz4XpIyD2a*7yRXdE*JEBDIwu7oj-qxLPPf>Er;HEYY&YbyNw=u z%t%i@xtaR*O$RnozkV6`HqtZC7-_%&BfdE;gC2j}NKZVGK?4V7kwz0o6DDM09OE!% z1_}z=fO9>Go_=~QO`MocFT9Wj+O;S@ow|2V#oUaAEbuAHN}-UDt@P=qU*OEo!MxE? zy?R>A<2a0K2KDK)jvO7e^ys6>G;7v22Y|5V;n}3gM$y^pT%R|69qeJS8&q2_^kdL7^H^hmRb^Uv<$j ziR|M~K2_3lcIF~Mvny}+9z6Eu@8hri2M#Lf8`C#&OvYv-7|G1qvK6Im+n&8cNl#pp z-7H__WwD3;B$ipKR?7)|2^8fPv3t1AA9_XCoA=D+jar%J9O;Sxl7l3yLInz-~9mr z5A-yT9}7O-K8>3+^~J(2L~5dH+N1K zzq<=@(N2gw9JNG!ekXFj23#a^yv^V`+*wY!7Z?qc$tex+p@dR7rAZS#E<`$20S%#}?JoZ67&VV@Vk9;=hum|fy^3B#T|9hijoalmlw!_T1ZK1laOeh78d3M$as0S?IPgz}oB9Uah~ zKEUI^0POeA0MFvg7z9~kDEeSH&d|50S+hBikCxyZT7|PU8ZwoZI&@&UA)dN+HQ+om z;2bqjj~)p)zYLJa4D`SQ2Hd0LCG_fLpobqeQ19LeII9zI&L-l#PQKGD&#vJtNL=BhG9SWR6UlJ2wm1O7rLE zK*rfiVPOZMBPak0A>S0z(xnBE6A#hy-UktuphF{0mxtnfkTkn4oR|Oa&jT1rXHsC>yJ`;`Z3zP`4cjk zPEc0XN!qgIbKnceXpH~Rwrz}461HzYMU3oh#%bEI<23Eu`6cD$eM$NGUqR+5rh^wX|Gnd zM$o7=ovO^yzf*g)hN(Gd)Bzp)_qjzK{X3yhnO6P_zHH}sqGRjw)KVVgc+y(a2Xv6x zwk%$LN7J9`R)MRh23aY(EUQM{p?~?lvQ@T{MXl~{S;Kg#2bDx?PxUBE`XN=)w|20k zAByoTEtN&3ve(iV9&T4l`XQqijt`I$HN!&QpwbKzUXfqav~W>qJxlt*1CLCQ(lmBC zLv+b9lRyXevEE#Y2BlzhmDY4)xj|fwf6j~ad>Cm+;aUhxU5fFs{;b$ zoUJt)4c?d|jV3_tX<2@9WK?Nb84+Ig9u*P3vNSp}QfEm&D>x`gW7~-^(iVsun%4dS>M&K5a|67Vg$8o08zLZe*Cs$FDh+&L z<4}xC8;7~9BNgH%QA>W}^5)3TV%(?rITw;tsFaM`UFox>5V+?_8S0)^`T_kU^$VU# z;CreE&>a#<4r zFNNOniTIb~d`!=XtpJ_G3v8nNeAzNDzsq^KPR;?G*#Z!G0}Zv$mNZHturyVe1yKb(h6) zex+Zn)n75c(he>_ssI0Se)tZQ^%Z;v%H{%m2&DFNB76yK_P^3!)p;uYf2}39UTna5 ztWv85c%Nwn^aYfCW-f0$wF@d_a{`(I-J3O=Fv-Vf>Io;O<-*Z1TzGlCC*0jbcOxwf z7zOkM)POU%G5m<0--J?sI;2c3y^kgZCn3AJYp5hgINVj%d~_cqb}b0t5mN z1I+=}1l9wp8@OK1;|DzK=l9xl506l(4)gx0>}lrxi}|s$i{!o#(4QSa-lf1;pbyXz z;4-V~4?EC!0}uN6gnZ!THC^x?ZEmYlpRFhJr3W9Y6+S*gSswtwTqeL}a~-yVhrj=8 zV?8|Ppp2-}dbyu%)n_YS9v?2l#U)HMZ91s{6)pe<09MZ$X2{n!FwoOeD`4Y7ofZ4o zR$aG>m;1xsK1%rcjX{|-18Bo_Sqt5|alO{gxm(pyRT`gjdg1TSywGx>v$eW84mLl~ zRMqaP(yj3mBb=QLTtCl+E?k$j02`%%jZ(ly30lkZITrnA5U^Pa*esD|Sx1b34O75| zX_a@_GzDy$qJ8_dTz@#w)mq&g2fM0(T@^f}Rj4Ccz|M+uE=R!5in(byC!+;yt^zh! z0h?>3e6Ya^*kA>0uv~uxz@`GL0XAU)o3MaQxKcT=5ewLeaaQLb%{)h7E5`he6R;Hv z*ou*6?l;(t1?XrC7qEFF{?_e>?Onk3E?|2Xu)PBtCGa6|B{ zLf&)%8$Ec)5U|l#SwDOT1bhetdx5%4XjvVK`Ujlx__67YfGqaE_qP0YP)0Y3-5Y;=Nkfk90L9(QeOC+2>6_!Ugo*>^~2u@HY|gHxcl6xo!2sZ$`i`M!+vdzy}6rqgKE#20UF5@S8!O z-MRYVhXa}K@2ekvLDi@gOzpnr5=HK73{@^z#;5R7XH;6q@E8sUM;5T^N_dobM zLiQ>U@OOl)V-oOpl=eUPI|}$a;`})w;O}T-|L3!(`sWY)HgVovkFV%Hp%!EaS=Vkw{Aw@JAK!M-}i#mGZ(Twe05Jvi)3lna&V;-B4Ef zUX%TF>@MPNcqhSn3Kqsk17&&&{Kn%p2i67fUECMw1`GyxT&4gc0M;9{0`SX}Ti%tC zd`AQ84p^tcdWqit{;$04;W7J^i_1c+!{wMav#>6L4k3;2nB1O!0qualyB(}=U_UW` zpbz+tWnNi#WY(vc^#!cof&QeVdGjD0=tBT*M@OI$P#?HcLE)SG@L4~}`b-u+Msg2%484n%i+XuqKV;1gAuWdpd^MTR8qd*Jb&*&<7ESmtm zyuGJ~dV9Zf-Pw5w_~!oPdwgy9wj2xPU0x&H4ll2Fd2SV8S9`pNCkV;x0HI?9$UNACd-i-8vb-Xp5rh7KK`^}#&Z(zY%4 z^IzH5)oC-2Yn-GPhCW<$?)+ptc;~gz7^qgj-hlm09@nbZWOaC|4Gs5OJm%BucuMP{ zeYKbe(Wp`Dk*A{0tu{RT**tpao>>1_m42ms%er<=KpyV@KU@Ew&+p|W>1QLrA&W%S z3f;T2Y!?r3-m1nynP*H_Rhz3r!}Fi(W?ES`1=BU+ze|?{=mrzT{P`y6lh#3=x0I_b z`&26)cpw2fJR|g@>7sdaWnZ_XF~0`}rios?cx-uYo8!&zI>^{Mv0y=#2n)-W?o&;g zFn!JV@7U2G?zu+?eb)`>Em>!5tv)TzPn{?#It9JDL1?u*MQUn(DXq1$K`#u3Jc&*o zSL^)y`o@XFhfj%P$Id}#eY8|BTr>VVcTT`Q93#%0`L8q%w)ltdJ8YcWM0WOZ=s~|n ze;IGde~T7+v~jO=N40(aX>lIwp&QS@7;F_jJ~f~JojMsHmuPYC`wjYN*0Wmgf9Ant zIxIGBEQJ1aKkm#QfwrO@gZo*FeoTO^CLTJ*O=9NE%{UXStreX#A0r}iAx~&=&%6Yk zE$dlp-v3WMm4vb0E#O;^{MPDXzT%)qPZU$9rb9QrRXp?%uX}54jl*3vMMOq!haNgh zL`UxsjT&+Oisfo?U(iC2yAl0w67Y+MOi}axZ{J>C`<2VLK4Jf`n}k%GAfn3V#NW_ndhCn&TBS*|bHaA)VUj`Pa%SXo0;rldiZF~H7oR?3r;^CfIprb^}V9J1u! z-M!}Pw|e{=vG1Rj>fgToH1>*(()*x6>#_H;T{{jst~ltKwBUQKlokYtf*9-jS1w3sRy?g6KOboBRLeZm#R?L{O6=z8a_Rb>M)b4KnSqIGX#Kk2M zNWgg%FG51LNc=BeeDoi|f4_e6TIIQq-;LGrU&HgOrt8;o{`35)>wok9e<$|Oy8f4r z1nJM;aemeHe_j9Mj!+ir_RqTWSCOIX_WxM;cUil?nC1Vv{qyeFKk>VL++VgqXQ${m zbYzazqz(=bFB16w52W|I z;r^JPFWThYgwc2ux|7ZDIsRJWf6tyXckKSVe*F>i z1TB3F3CV=sJR80{Mew2BRZeH1?0uF`*1@x$j%hj19W*{-9LcryFlSkwKO$3bq7+z!?SJ^geN%H1#G$MzJ?81mnr#8+`>PnOmvW5diP!j8)hP4kbwUu2KI9HOEtsAMNV^blb;H0 zb)HI|h7IFUW?Z>)xh~5xTC|Y#&*l7k%5(eA{lc%Iq@SVk4^cyTCbH8KKiP4KpLu8- bKj-4vr^4@3@k>|fcbIk08utTtDTn_BU7!%1 From e51970723b021c41f2c764b52e6b6ce3f921cac7 Mon Sep 17 00:00:00 2001 From: alawvt Date: Mon, 23 Aug 2021 17:00:23 -0400 Subject: [PATCH 10/23] Add IRUS patch (#766) --- dspace/config/dspace.cfg | 1 + dspace/config/hibernate.cfg.xml | 2 + dspace/config/launcher.xml | 7 + dspace/config/log4j.properties | 2 + dspace/config/modules/stats.cfg | 38 ++ .../config/spring/api/core-dao-services.xml | 1 + .../spring/api/core-factory-services.xml | 1 + dspace/config/spring/api/core-services.xml | 2 + dspace/config/spring/api/discovery.xml | 2 + .../spring/jspui/open-url-listeners.xml | 12 + .../spring/xmlui/open-url-listeners.xml | 12 + dspace/modules/additions/pom.xml | 7 + .../atmire-statistics-exporter-api/pom.xml | 34 ++ .../export/ExportUsageEventListener.java | 426 ++++++++++++++++++ .../statistics/export/OpenURLTracker.java | 82 ++++ .../OpenURLTrackerLoggerServiceImpl.java | 34 ++ .../export/RetryOpenUrlTracker.java | 63 +++ .../export/dao/OpenURLTrackerDAO.java | 13 + .../dao/impl/OpenURLTrackerDAOImpl.java | 17 + .../OpenURLTrackerLoggerServiceFactory.java | 18 + ...penURLTrackerLoggerServiceFactoryImpl.java | 18 + .../service/OpenURLTrackerLoggerService.java | 19 + .../statistics/util/SpiderDetector.java | 263 +++++++++++ .../usage/AbstractUsageEventListener.java | 70 +++ .../V6.0_2017.02.14__statistics-harvester.sql | 27 ++ .../V6.0_2017.02.14__statistics-harvester.sql | 8 + .../atmire-statistics-exporter/pom.xml | 24 + dspace/modules/jspui/pom.xml | 14 + dspace/modules/pom.xml | 1 + .../themes/vtmirage2/images/favicon_old.ico | Bin 0 -> 22486 bytes dspace/modules/xmlui/pom.xml | 8 + dspace/pom.xml | 38 +- dspace/src/main/config/build.xml | 22 + index.html | 1 + index.html.1 | 1 + 35 files changed, 1287 insertions(+), 1 deletion(-) create mode 100644 dspace/config/modules/stats.cfg create mode 100644 dspace/config/spring/jspui/open-url-listeners.xml create mode 100644 dspace/config/spring/xmlui/open-url-listeners.xml create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql create mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql create mode 100644 dspace/modules/atmire-statistics-exporter/pom.xml create mode 100644 dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/images/favicon_old.ico create mode 100644 index.html create mode 100644 index.html.1 diff --git a/dspace/config/dspace.cfg b/dspace/config/dspace.cfg index d17993d4d6..909a12d51d 100644 --- a/dspace/config/dspace.cfg +++ b/dspace/config/dspace.cfg @@ -2025,3 +2025,4 @@ include = ${module_dir}/translator.cfg include = ${module_dir}/usage-statistics.cfg include = ${module_dir}/versioning.cfg include = ${module_dir}/workflow.cfg +include = ${module_dir}/stats.cfg \ No newline at end of file diff --git a/dspace/config/hibernate.cfg.xml b/dspace/config/hibernate.cfg.xml index 26595653d8..2f66f1891a 100644 --- a/dspace/config/hibernate.cfg.xml +++ b/dspace/config/hibernate.cfg.xml @@ -80,5 +80,7 @@ --> + + diff --git a/dspace/config/launcher.xml b/dspace/config/launcher.xml index 138454c229..603bc6e229 100644 --- a/dspace/config/launcher.xml +++ b/dspace/config/launcher.xml @@ -380,4 +380,11 @@ org.dspace.app.util.Version + + retry-tracker + Retry all failed commits to the OpenURLTracker + + com.atmire.statistics.export.RetryOpenUrlTracker + + diff --git a/dspace/config/log4j.properties b/dspace/config/log4j.properties index 02231b3dbf..8beeddc013 100644 --- a/dspace/config/log4j.properties +++ b/dspace/config/log4j.properties @@ -115,3 +115,5 @@ log4j.logger.org.dspace.services=ERROR log4j.logger.org.dspace.servicemanager=ERROR log4j.logger.org.dspace.providers=ERROR log4j.logger.org.dspace.utils=ERROR + +log4j.logger.com.atmire.statistics.export.ExportUsageEventListener=DEBUG \ No newline at end of file diff --git a/dspace/config/modules/stats.cfg b/dspace/config/modules/stats.cfg new file mode 100644 index 0000000000..cea6a30f4f --- /dev/null +++ b/dspace/config/modules/stats.cfg @@ -0,0 +1,38 @@ + + +#-----------------------# +# Atmire stats exporter # +#-----------------------# + +# OPTIONAL metadata field used for filtering. +# If items with specific values for the "dc.type" field should be excluded, "dc.type" should be placed here. +# This should comply to the syntax schema.element.qualified or schema.element if the qualifier is null. +# stats.tracker.type-field = dc.type +# If "tracker.type-field" is set, the list of values must be defined in "tracker.type-value". +# This lists a comma separated list of values that will be excluded for the given field. +# stats.tracker.type-value = Article, Postprint + +# Enable the tracker +stats.tracker.enabled = true + +# Set the tracker environment to "test" or "production". Defaults to "test" if empty. +# The URL used by the test environment can be configured in property tracker.testurl +# The URL used by the production environment can be configured in property tracker.produrl +stats.tracker.environment = test +# The url used to test the submission of tracking info to. +stats.tracker.testurl = https://irus.jisc.ac.uk/counter/test/ +# The base url for submitting the tracking info to. +stats.tracker.produrl = https://irus.jisc.ac.uk/counter/ +# Identifies data as OpenURL 1.0 +stats.tracker.urlversion = Z39.88-2004 + +# The deployed user interface should be provided to build correct links to files. +# The dspace.type field can be set to either "xmlui" or "jspui". +stats.dspace.type = xmlui + +# Spider options +stats.spider.ipmatch.enabled = true +stats.spider.agentempty.enabled = false +stats.spider.agentregex.enabled = true +# Default is downloaded during build: ${dspace.dir}/config/COUNTER_Robots_list.txt +stats.spider.agentregex.regexfile = ${dspace.dir}/config/COUNTER_Robots_list.txt \ No newline at end of file diff --git a/dspace/config/spring/api/core-dao-services.xml b/dspace/config/spring/api/core-dao-services.xml index cc9244015e..85b8caf6c8 100644 --- a/dspace/config/spring/api/core-dao-services.xml +++ b/dspace/config/spring/api/core-dao-services.xml @@ -57,6 +57,7 @@ + diff --git a/dspace/config/spring/api/core-factory-services.xml b/dspace/config/spring/api/core-factory-services.xml index 0078c02524..ee15dcc2de 100644 --- a/dspace/config/spring/api/core-factory-services.xml +++ b/dspace/config/spring/api/core-factory-services.xml @@ -44,6 +44,7 @@ + diff --git a/dspace/config/spring/api/core-services.xml b/dspace/config/spring/api/core-services.xml index 2c6160f55f..26e38917c6 100644 --- a/dspace/config/spring/api/core-services.xml +++ b/dspace/config/spring/api/core-services.xml @@ -105,6 +105,8 @@ + + + + com.atmire + atmire-statistics-exporter-api + 1.0.0 + jar + + org.dspace dspace-api diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml new file mode 100644 index 0000000000..d4851a860b --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml @@ -0,0 +1,34 @@ + + + 4.0.0 + + + com.atmire + atmire-statistics-exporter + 1.0.0 + + + com.atmire + atmire-statistics-exporter-api + 1.0.0 + jar + Atmire statistics exporter API + atmire.com + + + + org.dspace + dspace-api + + + + + javax.servlet + servlet-api + 2.3 + provided + + + diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java new file mode 100644 index 0000000000..c240afdf7e --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java @@ -0,0 +1,426 @@ +/* + * ExportUsageEventListener.java + * + * Version: $Revision: 1 $ + * Date: $Date: 2010-04-09 11:01:28 +0200 (vr, 09 apr 2010) $ + * Copyright (c) @mire. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * - Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, + * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS + * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR + * TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE + * USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH + * DAMAGE. + */ +package com.atmire.statistics.export; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.UnsupportedEncodingException; +import java.net.HttpURLConnection; +import java.net.URL; +import java.net.URLConnection; +import java.net.URLEncoder; +import java.sql.SQLException; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.UUID; +import javax.servlet.http.HttpServletRequest; + +import com.atmire.statistics.export.factory.OpenURLTrackerLoggerServiceFactory; +import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; +import org.apache.commons.lang.StringUtils; +import org.apache.log4j.Logger; +import org.dspace.app.util.Util; +import org.dspace.content.Bitstream; +import org.dspace.content.Bundle; +import org.dspace.content.DCDate; +import org.dspace.content.Item; +import org.dspace.content.MetadataField; +import org.dspace.content.MetadataValue; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.MetadataFieldService; +import org.dspace.core.Context; +import org.dspace.core.LogManager; +import org.dspace.services.ConfigurationService; +import org.dspace.services.factory.DSpaceServicesFactory; +import org.dspace.services.model.Event; +import org.dspace.statistics.util.SpiderDetector; +import org.dspace.usage.AbstractUsageEventListener; +import org.dspace.usage.UsageEvent; + +/** + * User: kevin (kevin at atmire.com) + * Date: 30-mrt-2010 + * Time: 16:37:56 + */ +public class ExportUsageEventListener extends AbstractUsageEventListener { + /* Log4j logger*/ + private static Logger log = Logger.getLogger(ExportUsageEventListener.class); + + /* The metadata field which is to be checked for */ + private static MetadataField trackerType; + + /* A list of values the type might have */ + private static List trackerValues; + + /* The base url of the tracker */ + private static String baseUrl; + + private static String trackerUrlVersion; + + private static final String ITEM_VIEW = "Investigation"; + private static final String BITSTREAM_DOWNLOAD = "Request"; + + private static ConfigurationService configurationService; + + + public void init(Context context) { + try { + if (configurationService == null) { + configurationService = DSpaceServicesFactory.getInstance().getConfigurationService(); + } + if (trackerType == null) { + trackerType = resolveConfigPropertyToMetadataField(context,"tracker.type-field"); + + String[] metadataValues = configurationService.getArrayProperty("stats.tracker.type-value"); + if (metadataValues.length > 0) { + trackerValues = new ArrayList<>(); + for (String metadataValue : metadataValues) { + trackerValues.add(metadataValue.toLowerCase()); + } + } else { + trackerValues = null; + } + + if(StringUtils.equals(configurationService.getProperty("stats.tracker.environment"), "production")){ + baseUrl = configurationService.getProperty("stats.tracker.produrl"); + } + else { + baseUrl = configurationService.getProperty("stats.tracker.testurl"); + } + + trackerUrlVersion = configurationService.getProperty("stats.tracker.urlversion"); + + + } + } catch (Exception e) { + log.error("Unknown error resolving configuration for the export usage event.", e); + trackerType = null; + trackerValues = null; + baseUrl = null; + trackerUrlVersion = null; + } + } + + public void receiveEvent(Event event) { + if (event instanceof UsageEvent) { + UsageEvent ue = (UsageEvent) event; + Context context = ue.getContext(); + init(context); + boolean irusEnabled = configurationService.getBooleanProperty("stats.tracker.enabled"); + if (irusEnabled) { + try { + //Check for item investigation + if (ue.getObject() instanceof Item) { + Item item = (Item) ue.getObject(); + if(item.isArchived() && !ContentServiceFactory.getInstance().getItemService().canEdit(context, item)) { + init(context); + + if (shouldProcessItem(item)) { + processItem(ue.getContext(), item, null, ue.getRequest(), ITEM_VIEW); + } + } + } + //Check for bitstream download + if (ue.getObject() instanceof Bitstream) { + Bitstream bit = (Bitstream) ue.getObject(); + //Check for an item + if (0 < bit.getBundles().size()) { + if (!SpiderDetector.isSpider(ue.getRequest())) { + Bundle bundle = bit.getBundles().get(0); + if (bundle.getName() == null || !bundle.getName().equals("ORIGINAL")) + return; + + if (0 < bundle.getItems().size()) { + Item item = bundle.getItems().get(0); + + if (item.isArchived() && !ContentServiceFactory.getInstance().getItemService() + .canEdit(context, item)) { + //Check if we have a valid type of item ! + if (shouldProcessItem(item)) { + processItem(ue.getContext(), item, bit, ue.getRequest(), BITSTREAM_DOWNLOAD); + } + } + } + } else { + log.info("Robot (" + ue.getRequest().getHeader("user-agent") + ") accessed " + bit + .getName() + "/" + bit.getSource()); + } + } + } + } catch (Exception e) { + UUID id; + id = ue.getObject().getID(); + + int type; + try { + type = ue.getObject().getType(); + } catch (Exception e1) { + type = -1; + } + log.error(LogManager.getHeader(ue.getContext(), "Error while processing export of use event", + "Id: " + id + " type: " + type), e); + e.printStackTrace(); + } + } + } + } + + private boolean shouldProcessItem(Item item) { + if (trackerType != null && trackerValues != null) { + List types = ContentServiceFactory.getInstance().getItemService().getMetadata(item, trackerType.getMetadataSchema().getName(), trackerType.getElement(), trackerType.getQualifier(), Item.ANY); + + if (!types.isEmpty()) { + //Find out if we have a type that needs to be excluded + for (MetadataValue type : types) { + if (trackerValues.contains(type.getValue().toLowerCase())) { + //We have found no type so process this item + return false; + } + } + return true; + } else { + // No types in this item, so not excluded + return true; + } + } else { + // No types to be excluded + return true; + } + } + + private void processItem(Context context, Item item, Bitstream bitstream, HttpServletRequest request, String eventType) throws IOException, SQLException { + //We have a valid url collect the rest of the data + String clientIP = request.getRemoteAddr(); + if (configurationService.getBooleanProperty("useProxies", false) && request.getHeader("X-Forwarded-For") != null) { + /* This header is a comma delimited list */ + for (String xfip : request.getHeader("X-Forwarded-For").split(",")) { + /* proxy itself will sometime populate this header with the same value in + remote address. ordering in spec is vague, we'll just take the last + not equal to the proxy + */ + if (!request.getHeader("X-Forwarded-For").contains(clientIP)) { + clientIP = xfip.trim(); + } + } + } + String clientUA = StringUtils.defaultIfBlank(request.getHeader("USER-AGENT"), ""); + String referer = StringUtils.defaultIfBlank(request.getHeader("referer"), ""); + + //Start adding our data + StringBuilder data = new StringBuilder(); + data.append(URLEncoder.encode("url_ver", "UTF-8") + "=" + URLEncoder.encode(trackerUrlVersion, "UTF-8")); + data.append("&").append(URLEncoder.encode("req_id", "UTF-8")).append("=").append( URLEncoder.encode(clientIP, "UTF-8")); + data.append("&").append(URLEncoder.encode("req_dat", "UTF-8")).append("=").append( URLEncoder.encode(clientUA, "UTF-8")); + data.append("&").append(URLEncoder.encode("rft.artnum", "UTF-8")).append("=").append( URLEncoder.encode("oai:" + configurationService.getProperty("dspace.hostname") + ":" + item.getHandle(), "UTF-8")); + data.append("&").append(URLEncoder.encode("rfr_dat", "UTF-8")).append("=").append( URLEncoder.encode(referer, "UTF-8")); + data.append("&").append(URLEncoder.encode("rfr_id", "UTF-8")).append("=").append( URLEncoder.encode(configurationService.getProperty("dspace.hostname"), "UTF-8")); + data.append("&").append(URLEncoder.encode("url_tim", "UTF-8")).append("=").append( URLEncoder.encode(new DCDate(new Date()).toString(), "UTF-8")); + + if (BITSTREAM_DOWNLOAD.equals(eventType)) { + String bitstreamInfo = getBitstreamInfo(item, bitstream); + data.append("&").append( URLEncoder.encode("svc_dat", "UTF-8")).append("=").append( URLEncoder.encode(bitstreamInfo, "UTF-8")); + data.append("&").append( URLEncoder.encode("rft_dat", "UTF-8")).append("=").append( URLEncoder.encode(BITSTREAM_DOWNLOAD, "UTF-8")); + } else if (ITEM_VIEW.equals(eventType)) { + String itemInfo = getItemInfo(item); + data.append("&").append( URLEncoder.encode("svc_dat", "UTF-8")).append("=").append( URLEncoder.encode(itemInfo, "UTF-8")); + data.append("&").append( URLEncoder.encode("rft_dat", "UTF-8")).append("=").append( URLEncoder.encode(ITEM_VIEW, "UTF-8")); + } + + processUrl(context, baseUrl + "?" + data.toString()); + + } + + private String getBitstreamInfo(final Item item, final Bitstream bitstream) { + //only for jsp ui + // http://demo.dspace.org/jspui/handle/10673/2235 + // http://demo.dspace.org/jspui/bitstream/10673/2235/1/Captura.JPG + // + + + //only fror xmlui + // http://demo.dspace.org/xmlui/handle/10673/2235 + // http://demo.dspace.org/xmlui/bitstream/handle/10673/2235/Captura.JPG?sequence=1 + // + + String uiType = configurationService.getProperty("stats.dspace.type"); + StringBuilder sb = new StringBuilder(configurationService.getProperty("dspace.url")); + if ("jspui".equals(uiType)) { + + sb.append("/bitstream/").append(item.getHandle()).append("/").append(bitstream.getSequenceID()); + + // If we can, append the pretty name of the bitstream to the URL + try { + if (bitstream.getName() != null) { + sb.append("/").append(Util.encodeBitstreamName(bitstream.getName(), "UTF-8")); + } + } catch (UnsupportedEncodingException uee) { + // just ignore it, we don't have to have a pretty + // name at the end of the URL because the sequence id will + // locate it. However it means that links in this file might + // not work.... + } + + + } else { //xmlui + + String identifier = null; + if (item != null && item.getHandle() != null) { + identifier = "handle/" + item.getHandle(); + } else if (item != null) { + identifier = "item/" + item.getID(); + } else { + identifier = "id/" + bitstream.getID(); + } + + + sb.append("/bitstream/").append(identifier).append("/"); + + // If we can, append the pretty name of the bitstream to the URL + try { + if (bitstream.getName() != null) { + sb.append(Util.encodeBitstreamName(bitstream.getName(), "UTF-8")); + } + } catch (UnsupportedEncodingException uee) { + // just ignore it, we don't have to have a pretty + // name at the end of the URL because the sequence id will + // locate it. However it means that links in this file might + // not work.... + } + + sb.append("?sequence=").append(bitstream.getSequenceID()); + } + return sb.toString(); + } + + private String getItemInfo(final Item item) { + StringBuilder sb = new StringBuilder(configurationService.getProperty("dspace.url")); + sb.append("/handle/").append(item.getHandle()); + + return sb.toString(); + } + + + private static void processUrl(Context c, String urlStr) throws IOException, SQLException { + log.debug("Prepared to send url to tracker URL: " + urlStr); + System.out.println(urlStr); + URLConnection conn; + + try { + // Send data + URL url = new URL(urlStr); + conn = url.openConnection(); + + // Get the response + BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream())); + while (rd.readLine() != null) ; + + rd.close(); + if (((HttpURLConnection) conn).getResponseCode() != 200) { + ExportUsageEventListener.logfailed(c, urlStr); + } else if (log.isDebugEnabled()) { + log.debug("Successfully posted " + urlStr + " on " + new Date()); + } + } catch (Exception e) { + log.error("Failed to send url to tracker URL: " + urlStr); + ExportUsageEventListener.logfailed(c, urlStr); + } + } + + private static void tryReprocessFailed(Context context, OpenURLTracker tracker) throws SQLException { + boolean success = false; + URLConnection conn; + try { + URL url = new URL(tracker.getUrl()); + conn = url.openConnection(); + BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream())); + while (rd.readLine() != null) ; + rd.close(); + if (((HttpURLConnection) conn).getResponseCode() == 200) { + success = true; + } + } catch (Exception e) { + success = false; + } finally { + if (success) { + OpenURLTrackerLoggerServiceFactory.getInstance().getOpenUrlTrackerLoggerService().remove(context, tracker); + // If the tracker was able to post successfully, we remove it from the database + log.info("Successfully posted " + tracker.getUrl() + " from " + tracker.getUploadDate()); + } else { + // Still no luck - write an error msg but keep the entry in the table for future executions + log.error("Failed attempt from " + tracker.getUrl() + " originating from " + tracker.getUploadDate()); + } + } + } + + public static void reprocessFailedQueue(Context context) throws SQLException { + Context c = new Context(); + OpenURLTrackerLoggerServiceFactory instance = OpenURLTrackerLoggerServiceFactory.getInstance(); + if(instance==null){ + log.error("Error retrieving the \"OpenURLTrackerLoggerServiceFactory\" instance, aborting the processing"); + return; + } + OpenURLTrackerLoggerService openUrlTrackerLoggerService = instance.getOpenUrlTrackerLoggerService(); + if(openUrlTrackerLoggerService==null){ + log.error("Error retrieving the \"openUrlTrackerLoggerService\" instance, aborting the processing"); + return; + } + List openURLTrackers = openUrlTrackerLoggerService.findAll(c); + for(OpenURLTracker openURLTracker : openURLTrackers){ + ExportUsageEventListener.tryReprocessFailed(context, openURLTracker) ; + } + + try { + c.abort(); + } catch (Exception ignored) { + } + } + + public static void logfailed(Context context, String url) throws SQLException { + Date now = new Date(); + if (url.equals("")) return; + OpenURLTrackerLoggerService service = OpenURLTrackerLoggerServiceFactory.getInstance().getOpenUrlTrackerLoggerService(); + OpenURLTracker tracker = service.create(context); + tracker.setUploadDate(now); + tracker.setUrl(url); + // TODO service tracker update + } + + private static MetadataField resolveConfigPropertyToMetadataField(Context context, String fieldName) throws SQLException { + String metadataField = configurationService.getProperty("stats." + fieldName); + if (metadataField != null && 0 < metadataField.trim().length()) { + metadataField = metadataField.trim(); + MetadataFieldService metadataFieldService = ContentServiceFactory.getInstance().getMetadataFieldService(); + return metadataFieldService.findByElement(context,metadataField.split("\\.")[0],metadataField.split("\\.")[1],metadataField.split("\\.").length == 2 ? null : metadataField.split("\\.")[2]); + } + return null; + } +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java new file mode 100644 index 0000000000..dbc7667fc5 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java @@ -0,0 +1,82 @@ +package com.atmire.statistics.export; + +import org.dspace.core.ReloadableEntity; +import org.hibernate.proxy.HibernateProxyHelper; + +import javax.persistence.*; +import java.util.Date; + +/** + * Created by jonas - jonas@atmire.com on 09/02/17. + */ +@Entity +@Table(name="OpenUrlTracker") +public class OpenURLTracker implements ReloadableEntity { + + @Id + @Column(name="tracker_id") + @GeneratedValue(strategy = GenerationType.SEQUENCE ,generator="openurltracker_seq") + @SequenceGenerator(name="openurltracker_seq", sequenceName="openurltracker_seq", allocationSize = 1) + private Integer id; + + @Column(name = "tracker_url", length = 1000) + private String url; + + @Column(name = "uploaddate") + @Temporal(TemporalType.DATE) + private Date uploadDate; + + protected OpenURLTracker(){ + + } + + @Override + public Integer getID() { + return id; + } + + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public Date getUploadDate() { + return uploadDate; + } + + public void setUploadDate(Date uploadDate) { + this.uploadDate = uploadDate; + } + + + @Override + public boolean equals(Object o) { + if (this == o) + { + return true; + } + Class objClass = HibernateProxyHelper.getClassWithoutInitializingProxy(o); + if (getClass() != objClass) + { + return false; + } + + final OpenURLTracker that = (OpenURLTracker)o; + if (this.getID() != that.getID()) + { + return false; + } + + return true; + } + @Override + public int hashCode() { + int hash=8; + hash=74*hash+ this.getID(); + return hash; + } +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java new file mode 100644 index 0000000000..707d6fb703 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java @@ -0,0 +1,34 @@ +package com.atmire.statistics.export; + +import com.atmire.statistics.export.dao.OpenURLTrackerDAO; +import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; +import org.dspace.core.Context; +import org.springframework.beans.factory.annotation.Autowired; + +import java.sql.SQLException; +import java.util.List; + +/** + * Created by jonas - jonas@atmire.com on 09/02/17. + */ +public class OpenURLTrackerLoggerServiceImpl implements OpenURLTrackerLoggerService { + + @Autowired(required = true) + protected OpenURLTrackerDAO openURLTrackerDAO; + + @Override + public void remove(Context context, OpenURLTracker openURLTracker) throws SQLException { + openURLTrackerDAO.delete(context,openURLTracker); + } + + @Override + public List findAll(Context context) throws SQLException { + return openURLTrackerDAO.findAll(context,OpenURLTracker.class); + } + + @Override + public OpenURLTracker create(Context context) throws SQLException { + OpenURLTracker openURLTracker = openURLTrackerDAO.create(context, new OpenURLTracker()); + return openURLTracker; + } +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java new file mode 100644 index 0000000000..758c4d85a3 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java @@ -0,0 +1,63 @@ +package com.atmire.statistics.export; + +import org.apache.commons.cli.*; +import org.apache.log4j.Logger; +import org.dspace.core.Context; + +import java.sql.SQLException; + +/** + * Created by IntelliJ IDEA. + * Date: 25/04/12 + * Time: 09:32 + * + * @author: Kevin Van Ransbeeck (kevin van ransbeeck @ atmire dot com) + */ +public class RetryOpenUrlTracker { + private static final Logger log = Logger.getLogger(RetryOpenUrlTracker.class); + + /* Command Line execution */ + public static void main(String[] args) throws SQLException { + Context context = new Context(); + + context.turnOffAuthorisationSystem(); + + String usage = "com.atmire.statistics.export.RetryOpenUrlTracker [-a ]] or nothing to retry all failed attempts."; + Options options = new Options(); + HelpFormatter formatter = new HelpFormatter(); + CommandLine line = null; + + OptionBuilder.withArgName("Open URL Tracker"); + OptionBuilder.hasArg(true); + OptionBuilder.withDescription("Add a new row to the table (test purposes only)"); + options.addOption(OptionBuilder.create("a")); + OptionBuilder.isRequired(false); + OptionBuilder.withDescription("print this help message"); + options.addOption(OptionBuilder.create("h")); + + try { + line = new PosixParser().parse(options, args); + } catch (Exception e) { + // automatically generate the help statement + formatter.printHelp(usage, e.getMessage(), options, ""); + System.exit(1); + } + if (line.hasOption("h")) { + // automatically generate the help statement + formatter.printHelp(usage, options); + System.exit(1); + } + + if (line.hasOption("a")) { + ExportUsageEventListener.logfailed(context, line.getOptionValue("a")); + log.info("Created dummy entry in OpenUrlTracker with URL: " + line.getOptionValue("a")); + } else { + ExportUsageEventListener.reprocessFailedQueue(context); + } + context.restoreAuthSystemState(); + try { + context.complete(); + } catch (Exception ignored) { + } + } +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java new file mode 100644 index 0000000000..6218f2bfe1 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java @@ -0,0 +1,13 @@ +package com.atmire.statistics.export.dao; + + +import com.atmire.statistics.export.OpenURLTracker; +import org.dspace.core.GenericDAO; + +/** + * Created by jonas - jonas@atmire.com on 09/02/17. + */ +public interface OpenURLTrackerDAO extends GenericDAO { + + +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java new file mode 100644 index 0000000000..71619dbac4 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java @@ -0,0 +1,17 @@ +package com.atmire.statistics.export.dao.impl; + +import com.atmire.statistics.export.OpenURLTracker; +import com.atmire.statistics.export.dao.OpenURLTrackerDAO; +import org.dspace.core.AbstractHibernateDAO; + +/** + * Created by jonas - jonas@atmire.com on 09/02/17. + */ +public class OpenURLTrackerDAOImpl extends AbstractHibernateDAO implements OpenURLTrackerDAO { + + + protected OpenURLTrackerDAOImpl(){ + super(); + } + +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java new file mode 100644 index 0000000000..422aea6972 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java @@ -0,0 +1,18 @@ +package com.atmire.statistics.export.factory; + +import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; +import org.dspace.services.factory.DSpaceServicesFactory; + +/** + * Created by jonas - jonas@atmire.com on 09/02/17. + */ +public abstract class OpenURLTrackerLoggerServiceFactory { + + public abstract OpenURLTrackerLoggerService getOpenUrlTrackerLoggerService(); + + public static OpenURLTrackerLoggerServiceFactory getInstance(){ + return DSpaceServicesFactory.getInstance().getServiceManager().getServiceByName("openURLTrackerLoggerServiceFactory", OpenURLTrackerLoggerServiceFactory.class); + + } + +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java new file mode 100644 index 0000000000..48362f708e --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java @@ -0,0 +1,18 @@ +package com.atmire.statistics.export.factory; + +import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; +import org.springframework.beans.factory.annotation.Autowired; + +/** + * Created by jonas - jonas@atmire.com on 09/02/17. + */ +public class OpenURLTrackerLoggerServiceFactoryImpl extends OpenURLTrackerLoggerServiceFactory{ + + @Autowired(required = true) + private OpenURLTrackerLoggerService openURLTrackerLoggerService; + + @Override + public OpenURLTrackerLoggerService getOpenUrlTrackerLoggerService() { + return openURLTrackerLoggerService; + } +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java new file mode 100644 index 0000000000..8476618ccb --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java @@ -0,0 +1,19 @@ +package com.atmire.statistics.export.service; + +import com.atmire.statistics.export.OpenURLTracker; +import org.dspace.core.Context; + +import java.sql.SQLException; +import java.util.List; + +/** + * Created by jonas - jonas@atmire.com on 09/02/17. + */ +public interface OpenURLTrackerLoggerService { + + void remove(Context context, OpenURLTracker openURLTracker) throws SQLException; + + List findAll(Context context) throws SQLException; + + OpenURLTracker create(Context context) throws SQLException; +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java new file mode 100644 index 0000000000..b7ec11fbfa --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java @@ -0,0 +1,263 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.statistics.util; + +import java.io.*; +import java.util.*; +import java.util.regex.*; +import javax.servlet.http.*; +import org.apache.log4j.*; +import org.dspace.services.factory.*; +import org.dspace.statistics.factory.*; + +/** + * SpiderDetector is used to find IP's that are spiders... + * In future someone may add UserAgents and Host Domains + * to the detection criteria here. + * + * @author kevinvandevelde at atmire.com + * @author ben at atmire.com + * @author Mark Diggory (mdiggory at atmire.com) + * @author Kevin Van Ransbeeck at atmire.com + */ +public class SpiderDetector { + + private static final Logger log = Logger.getLogger(SpiderDetector.class); + + + + /** + * Sparse HAshTable structure to hold IP Address Ranges. + */ + private static IPTable table = null; + private static Set spidersRegex = Collections.synchronizedSet(new HashSet()); + private static Set spidersMatched = null; + + /** + * Utility method which Reads the ip addresses out a file & returns them in a Set + * + * @param spiderIpFile the location of our spider file + * @return a vector full of ip's + * @throws java.io.IOException could not happen since we check the file be4 we use it + */ + public static Set readIpAddresses(File spiderIpFile) throws IOException { + Set ips = new HashSet<>(); + + if (!spiderIpFile.exists() || !spiderIpFile.isFile()) { + return ips; + } + + //Read our file & get all them ip's + try (BufferedReader in = new BufferedReader(new FileReader(spiderIpFile))){ + String line; + while ((line = in.readLine()) != null) { + if (!line.startsWith("#")) { + line = line.trim(); + + if (!line.equals("") && !Character.isDigit(line.charAt(0))) { + // is a hostname + // add this functionality later... + } else if (!line.equals("")) { + ips.add(line); + // is full v4 ip (too tired to deal with v6)... + } + } else { + // ua.add(line.replaceFirst("#","").replaceFirst("UA","").trim()); + // ... add this functionality later + } + } + } + return ips; + } + + /** + * Get an immutable Set representing all the Spider Addresses here + * + * @return Set setOfIpAddresses + */ + public static Set getSpiderIpAddresses() { + loadSpiderIpAddresses(); + return table.toSet(); + } + + /* + private loader to populate the table from files. + */ + + private static synchronized void loadSpiderIpAddresses() { + if (table == null) { + table = new IPTable(); + + String filePath = DSpaceServicesFactory.getInstance().getConfigurationService().getProperty("dspace.dir"); + try { + File spidersDir = new File(filePath, "config/spiders"); + + if (spidersDir.exists() && spidersDir.isDirectory()) { + for (File file : spidersDir.listFiles()) { + for (String ip : readIpAddresses(file)) { + table.add(ip); + } + log.info("Loaded Spider IP file: " + file); + } + } else { + log.info("No spider file loaded"); + } + } catch (Exception e) { + log.error("Error Loading Spiders:" + e.getMessage(), e); + } + } + } + + /** + * Static Service Method for testing spiders against existing spider files. + *

+ * In the future this will be extended to support User Agent and + * domain Name detection. + *

+ * In future spiders HashSet may be optimized as byte offset array to + * improve performance and memory footprint further. + * + * @param request + * @return true|false if the request was detected to be from a spider + */ + public static boolean isSpider(HttpServletRequest request) { + /* + * 1) If the IP address matches the spider IP addresses (this is the current implementation) + */ + boolean checkSpidersIP = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.ipmatch.enabled", true, true); + if (checkSpidersIP) { + if (StatisticsServiceFactory.getInstance().getSolrLoggerService().isUseProxies() && request.getHeader("X-Forwarded-For") != null) { + /* This header is a comma delimited list */ + for (String xfip : request.getHeader("X-Forwarded-For").split(",")) { + if (isSpider(xfip)) { + log.debug("spider.ipmatch"); + return true; + } + } + } else if (isSpider(request.getRemoteAddr())) { + log.debug("spider.ipmatch"); + return true; + } + } + /* + * 2) if the user-agent header is empty - DISABLED BY DEFAULT - + */ + boolean checkSpidersEmptyAgent = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.agentempty.enabled", false, true); + if (checkSpidersEmptyAgent) { + if (request.getHeader("user-agent") == null || request.getHeader("user-agent").length() == 0) { + log.debug("spider.agentempty"); + return true; + } + } + /* + * 3) if the user-agent corresponds to one of the regexes at http://www.projectcounter.org/r4/COUNTER_robot_txt_list_Jan_2011.txt + */ + boolean checkSpidersTxt = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.agentregex.enabled", true, true); + if (checkSpidersTxt) { + String userAgent = request.getHeader("user-agent"); + + if (userAgent != null && !userAgent.equals("")) { + return isSpiderRegex(userAgent); + } + } + return false; + } + + /** + * Check individual IP is a spider. + * + * @param ip + * @return if is spider IP + */ + public static boolean isSpider(String ip) { + if (table == null) { + SpiderDetector.loadSpiderIpAddresses(); + } + + try { + if (table.contains(ip)) { + return true; + } + } catch (Exception e) { + return false; + } + + return false; + } + + /** + * Checks the user-agent string vs a set of known regexes from spiders + * A second Set is kept for fast-matching. + * If a user-agent is matched once, it is added to this set with "known agents". + * If this user-agent comes back later, we can do a quick lookup in this set, + * instead of having to loop over the entire set with regexes again. + * + * @param userAgent String + * @return true if the user-agent matches a regex + */ + public static boolean isSpiderRegex(String userAgent) { + if (spidersMatched != null && spidersMatched.contains(userAgent)) { + log.debug("spider.agentregex"); + return true; + } else { + synchronized(spidersRegex) { + if (spidersRegex.isEmpty()) + loadSpiderRegexFromFile(); + } + + if (spidersRegex != null) { + for (Object regex : spidersRegex.toArray()) { + Matcher matcher = ((Pattern) regex).matcher(userAgent); + if (matcher.find()) { + if (spidersMatched == null) { + spidersMatched = new HashSet<>(); + } + if (spidersMatched.size() >= 100) { + spidersMatched.clear(); + } + spidersMatched.add(userAgent); + log.debug("spider.agentregex"); + return true; + } + } + } + return false; + } + } + + /** + * Populate static Set spidersRegex from local txt file. + * Original file downloaded from http://www.projectcounter.org/r4/COUNTER_robot_txt_list_Jan_2011.txt during build + */ + public static void loadSpiderRegexFromFile() { + String spidersTxt = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.agentregex.regexfile", String.class); + DataInputStream in = null; + try { + FileInputStream fstream = new FileInputStream(spidersTxt); + in = new DataInputStream(fstream); + BufferedReader br = new BufferedReader(new InputStreamReader(in)); + String strLine; + while ((strLine = br.readLine()) != null) { + spidersRegex.add(Pattern.compile(strLine, Pattern.CASE_INSENSITIVE)); + } + log.info("Loaded Spider Regex file: " + spidersTxt); + } catch (FileNotFoundException e) { + log.error("File with spiders regex not found @ " + spidersTxt); + } catch (IOException e) { + log.error("Could not read from file " + spidersTxt); + } finally { + try { + if (in != null) { + in.close(); + } + } catch (IOException e) { + log.error("Could not close file " + spidersTxt); + } + } + } +} \ No newline at end of file diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java new file mode 100644 index 0000000000..e6cb685fc8 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java @@ -0,0 +1,70 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.usage; + +import org.dspace.services.EventService; +import org.dspace.services.model.EventListener; +import org.springframework.beans.BeansException; +import org.springframework.beans.factory.config.BeanPostProcessor; + +/** + * AbstractUsageEventListener is used as the base class for listening events running + * in the EventService. + * + * @author Mark Diggory (mdiggory at atmire.com) + * @version $Revision: $ + */ +public abstract class AbstractUsageEventListener implements EventListener, BeanPostProcessor { + + public AbstractUsageEventListener() { + super(); + } + + @Override + public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException { + return bean; + } + + @Override + public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException { + if(beanName.equals("org.dspace.services.EventService")) + { + setEventService((EventService) bean); + } + return bean; + } + + /** + * Empty String[] flags to have Listener + * consume any event name prefixes. + */ + public String[] getEventNamePrefixes() { + return new String[0]; + } + + /** + * Currently consumes events generated for + * all resources. + */ + public String getResourcePrefix() { + return null; + } + + public void setEventService(EventService service) { + if(service != null) + { + service.registerEventListener(this); + } + else + { + throw new IllegalStateException("EventService handed to Listener cannot be null"); + } + + } + +} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql new file mode 100644 index 0000000000..dd0d67b187 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql @@ -0,0 +1,27 @@ +-- CREATE SEQUENCE "openurltracker_seq" start with 1 increment by 1 nocache nocycle noorder; +-- +-- CREATE table "OpenUrlTracker" ( +-- "tracker_id" NUMBER NOT NULL, +-- "tracker_url" VARCHAR2(255) NOT NULL, +-- "uploaddate" DATE, +-- constraint "OpenUrlTracker_PK" primary key ("tracker_id") +-- ); +-- +-- CREATE trigger "BI_OpenUrlTracker" +-- before insert on "OpenUrlTracker" +-- for each row +-- begin +-- if :NEW."tracker_id" is null then +-- select "openurltracker_seq".nextval into :NEW."tracker_id" from dual; +-- end if; +-- end; + +CREATE SEQUENCE openurltracker_seq; + +CREATE TABLE OpenUrlTracker +( + tracker_id NUMBER, + tracker_url VARCHAR2(1000), + uploaddate DATE, + CONSTRAINT OpenUrlTracker_PK PRIMARY KEY (tracker_id) +); diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql new file mode 100644 index 0000000000..21b9d7401d --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql @@ -0,0 +1,8 @@ +CREATE SEQUENCE openurltracker_seq; + +CREATE TABLE OpenUrlTracker +( + tracker_id INTEGER PRIMARY KEY, + tracker_url VARCHAR(1000), + uploaddate DATE +); diff --git a/dspace/modules/atmire-statistics-exporter/pom.xml b/dspace/modules/atmire-statistics-exporter/pom.xml new file mode 100644 index 0000000000..bcff958db7 --- /dev/null +++ b/dspace/modules/atmire-statistics-exporter/pom.xml @@ -0,0 +1,24 @@ + + + 4.0.0 + + com.atmire + atmire-statistics-exporter + pom + atmire.com + Atmire statistics exporter + 1.0.0 + + + org.dspace + modules + [6.0,6.10] + + + + atmire-statistics-exporter-api + + + diff --git a/dspace/modules/jspui/pom.xml b/dspace/modules/jspui/pom.xml index 90a0103c26..9f94969fb2 100644 --- a/dspace/modules/jspui/pom.xml +++ b/dspace/modules/jspui/pom.xml @@ -89,6 +89,13 @@ org.dspace dspace-jspui war + + + + WEB-INF/classes/** + @@ -103,6 +110,13 @@ + + com.atmire + atmire-statistics-exporter-api + 1.0.0 + jar + + org.dspace.modules additions diff --git a/dspace/modules/pom.xml b/dspace/modules/pom.xml index 32668031ae..8406dab7ce 100644 --- a/dspace/modules/pom.xml +++ b/dspace/modules/pom.xml @@ -25,6 +25,7 @@ as a dependency in most other modules in [src]/dspace/modules --> additions + atmire-statistics-exporter + + com.atmire + atmire-statistics-exporter-api + 1.0.0 + jar + + org.dspace diff --git a/dspace/pom.xml b/dspace/pom.xml index 5a8c643b98..de87f5fedd 100644 --- a/dspace/pom.xml +++ b/dspace/pom.xml @@ -40,7 +40,7 @@ modules - + + + ${basedir} + true + + config/default.context.xml + config/dspace.cfg + config/log4j-handle-plugin.properties + config/log4j.properties + config/modules/curate.cfg + config/modules/com.atmire.statistics.oai.cfg + config/modules/solr-statistics.cfg + + + + + ${basedir} + false + + config/default.context.xml + config/dspace.cfg + config/log4j-handle-plugin.properties + config/log4j.properties + config/modules/curate.cfg + config/modules/com.atmire.statistics.oai.cfg + config/modules/solr-statistics.cfg + + + @@ -168,6 +198,12 @@ + + + com.atmire + atmire-statistics-exporter-api + 1.0.0 + com.lyncode builder-commons diff --git a/dspace/src/main/config/build.xml b/dspace/src/main/config/build.xml index b785818073..b4848ab168 100644 --- a/dspace/src/main/config/build.xml +++ b/dspace/src/main/config/build.xml @@ -122,6 +122,7 @@ Common usage: + @@ -183,6 +184,7 @@ Common usage: + @@ -862,6 +864,8 @@ Common usage: + + ==================================================================== The DSpace code has been installed. @@ -947,6 +951,24 @@ You may manually install this file by following these steps: + + + Downloading: https://raw.githubusercontent.com/atmire/COUNTER-Robots/master/generated/COUNTER_Robots_list.txt + + + + + + + + + + + + + + + diff --git a/index.html b/index.html new file mode 100644 index 0000000000..15f23b8419 --- /dev/null +++ b/index.html @@ -0,0 +1 @@ +

IRUS Test Tracker

\ No newline at end of file diff --git a/index.html.1 b/index.html.1 new file mode 100644 index 0000000000..5ecd536e90 --- /dev/null +++ b/index.html.1 @@ -0,0 +1 @@ +

IRUS Tracker

\ No newline at end of file From 03b74cb7154bf84a0761a4acc310ae1040ab55b8 Mon Sep 17 00:00:00 2001 From: "Anne S. Lawrence" Date: Tue, 24 Aug 2021 09:50:21 -0400 Subject: [PATCH 11/23] Add IRUS stats URL --- dspace/config/modules/stats.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dspace/config/modules/stats.cfg b/dspace/config/modules/stats.cfg index cea6a30f4f..9312e85fec 100644 --- a/dspace/config/modules/stats.cfg +++ b/dspace/config/modules/stats.cfg @@ -22,7 +22,7 @@ stats.tracker.environment = test # The url used to test the submission of tracking info to. stats.tracker.testurl = https://irus.jisc.ac.uk/counter/test/ # The base url for submitting the tracking info to. -stats.tracker.produrl = https://irus.jisc.ac.uk/counter/ +stats.tracker.produrl = https://irus.jisc.ac.uk/counter/us # Identifies data as OpenURL 1.0 stats.tracker.urlversion = Z39.88-2004 From 987b1aedf3bfed13bfa5536ca1a414a1c027bcba Mon Sep 17 00:00:00 2001 From: alawvt Date: Thu, 26 Aug 2021 11:34:43 -0400 Subject: [PATCH 12/23] Theme color (#770) * update VT brand colors * update link colors to match VT News link colors * change links in news-xmlui.xml from bold to underline --- dspace/config/news-xmlui.xml | 8 ++++---- .../src/main/webapp/themes/vtmirage2/styles/_style.scss | 4 ++++ .../_bootstrap_variables.scss | 9 +++++---- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/dspace/config/news-xmlui.xml b/dspace/config/news-xmlui.xml index 12b4f1ee95..f105a9d770 100644 --- a/dspace/config/news-xmlui.xml +++ b/dspace/config/news-xmlui.xml @@ -3,13 +3,13 @@
VTechWorks -

VTechWorks provides global access to Virginia Tech scholarship. VTechWorks includes journal articles, books, theses, dissertations, conference papers, slide presentations, technical reports, working papers, administrative documents, videos, images, and more by faculty, students, and staff. Write vtechworks@vt.edu to get help adding your content to VTechWorks. Read an article about VTechWorks or visit the Open@VT blog to learn about current VTechWorks activities.

+

VTechWorks provides global access to Virginia Tech scholarship. VTechWorks includes journal articles, books, theses, dissertations, conference papers, slide presentations, technical reports, working papers, administrative documents, videos, images, and more by faculty, students, and staff. Write vtechworks@vt.edu to get help adding your content to VTechWorks. Read an article about VTechWorks or visit the Open@VT blog to learn about current VTechWorks activities.

-

Want to publish a standalone dataset? Visit Virginia Tech Data Repository.

+

Want to publish a standalone dataset? Visit Virginia Tech Data Repository.

-

Faculty can deposit items to VTechWorks from Elements (EFARs). Visit the Provost's Elements page to learn more and to log in to Elements.

+

Faculty can deposit items to VTechWorks from Elements (EFARs). Visit the Provost's Elements page to learn more and to log in to Elements.

-

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

+

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

diff --git a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/_style.scss b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/_style.scss index 953079fe1e..0f7e03b071 100644 --- a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/_style.scss +++ b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/_style.scss @@ -69,6 +69,10 @@ width:200px; font-weight: 900; } +.a-ul { +text-decoration: underline; +} + // Altmetric styles .altmetric-embed { padding-top: 25px; diff --git a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/classic_mirage_color_scheme/_bootstrap_variables.scss b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/classic_mirage_color_scheme/_bootstrap_variables.scss index 3a9b9e5c47..53b4243c78 100644 --- a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/classic_mirage_color_scheme/_bootstrap_variables.scss +++ b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/styles/classic_mirage_color_scheme/_bootstrap_variables.scss @@ -10,15 +10,16 @@ // ---------------------- // VT Colors -$chicago-maroon: #660000; -$burnt-orange: #ff6600; +$chicago-maroon: #861F41; +$burnt-orange: #C64600; +$vt-blue-4t: #668AAA; // Web accessibility 2.0 compatible colors $gray-on-almost-white-aim: #494949; $gray-on-white-aim: #4c4c4c; -$content-link: #8c5206; -$content-link-hover: #12252c; +$content-link: $burnt-orange; +$content-link-hover: $vt-blue-4t; $content-text: #000000; $brand-primary: $chicago-maroon; From 3e8984c5070b066386ac760e4ec3fa59b178f696 Mon Sep 17 00:00:00 2001 From: alawvt Date: Wed, 8 Sep 2021 15:46:36 -0400 Subject: [PATCH 13/23] add SWORD to bundle list (#772) Resolves #771 --- dspace/config/local.cfg | 2 +- dspace/modules/xmlui/src/main/webapp/i18n/messages.xml | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/dspace/config/local.cfg b/dspace/config/local.cfg index c0aa44e976..a060ea665e 100644 --- a/dspace/config/local.cfg +++ b/dspace/config/local.cfg @@ -352,7 +352,7 @@ xmlui.theme.mirage.item-list.emphasis = file # have the appropriate privileges (add & write) on the bundle then that bundle will # not be shown to the user as an option. # Allow upload to MOVIEPOSTER bundle from web UI -xmlui.bundle.upload = ORIGINAL, METADATA, THUMBNAIL, MOVIEPOSTER, LICENSE, CC-LICENSE +xmlui.bundle.upload = ORIGINAL, METADATA, THUMBNAIL, MOVIEPOSTER, LICENSE, CC-LICENSE, SWORD ########################################################################################## # Overwritten modules files: Variable in config/modules files overwritten by local.cfg, # diff --git a/dspace/modules/xmlui/src/main/webapp/i18n/messages.xml b/dspace/modules/xmlui/src/main/webapp/i18n/messages.xml index bd193b85b6..832bfb18bd 100644 --- a/dspace/modules/xmlui/src/main/webapp/i18n/messages.xml +++ b/dspace/modules/xmlui/src/main/webapp/i18n/messages.xml @@ -1586,8 +1586,9 @@ Metadata Files Thumbnails Licenses - Movie Posters + Movie Posters Creative Commons Licenses + SWORD File Please enter the name of the file on your computer corresponding to your item. If you click "Browse...", a new window will appear in which you can locate and select the file from your computer. Description From 8cf231113c120199a8c9649ef9bde6123febfb7c Mon Sep 17 00:00:00 2001 From: alawvt Date: Mon, 13 Dec 2021 15:21:28 -0500 Subject: [PATCH 14/23] Add Matomo script (#778) --- .../vtmirage2/xsl/core/page-structure.xsl | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl index ce6e838926..e2369bd02b 100644 --- a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl +++ b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl @@ -331,6 +331,25 @@ + + + + + From 85b3679c25841c2ee36e812ba136151e0905e52c Mon Sep 17 00:00:00 2001 From: alawvt Date: Thu, 16 Dec 2021 16:51:48 -0500 Subject: [PATCH 15/23] Add patches for DS-4291 and DS-4034, fix author browse (#779) * copy existing files to modules for DS-4034 #2276 * copy existing files to modules for DS-4291 #2463 * make changes in DS-4034 #2276 and DS-4291 #2463to fix author browse --- .../java/org/dspace/browse/BrowseEngine.java | 804 +++++++++++++++ .../java/org/dspace/browse/BrowseInfo.java | 927 ++++++++++++++++++ .../java/org/dspace/browse/SolrBrowseDAO.java | 837 ++++++++++++++++ .../org/dspace/sort/OrderFormatAuthor.java | 28 + .../java/org/dspace/sort/OrderFormatText.java | 28 + .../artifactbrowser/ConfigurableBrowse.java | 12 +- 6 files changed, 2631 insertions(+), 5 deletions(-) create mode 100644 dspace/modules/additions/src/main/java/org/dspace/browse/BrowseEngine.java create mode 100644 dspace/modules/additions/src/main/java/org/dspace/browse/BrowseInfo.java create mode 100644 dspace/modules/additions/src/main/java/org/dspace/browse/SolrBrowseDAO.java create mode 100644 dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatAuthor.java create mode 100644 dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatText.java diff --git a/dspace/modules/additions/src/main/java/org/dspace/browse/BrowseEngine.java b/dspace/modules/additions/src/main/java/org/dspace/browse/BrowseEngine.java new file mode 100644 index 0000000000..fe4738c0c7 --- /dev/null +++ b/dspace/modules/additions/src/main/java/org/dspace/browse/BrowseEngine.java @@ -0,0 +1,804 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.browse; + +import org.apache.log4j.Logger; +import org.dspace.content.Collection; +import org.dspace.content.Community; +import org.dspace.content.Item; +import org.dspace.core.Context; +import org.dspace.core.LogManager; +import org.dspace.sort.OrderFormat; +import org.dspace.sort.SortOption; + +import java.sql.SQLException; +import java.util.ArrayList; +import java.util.List; + +/** + * This class does most of the actual grunt work of preparing a browse + * result. It takes in to a couple of available methods (depending on your + * desired browse type) a BrowserScope object, and uses this to produce a + * BrowseInfo object which is sufficient to describe to the User Interface + * the results of the requested browse + * + * @author Richard Jones + * + */ +public class BrowseEngine +{ + /** the logger for this class */ + private static final Logger log = Logger.getLogger(BrowseEngine.class); + + /** the browse scope which is the basis for our browse */ + private BrowserScope scope; + + /** the DSpace context */ + private final Context context; + + /** The Data Access Object for the browse tables */ + private final BrowseDAO dao; + + /** The Browse Index associated with the Browse Scope */ + private BrowseIndex browseIndex; + + /** + * Create a new instance of the Browse engine, using the given DSpace + * Context object. This will automatically assign a Data Access Object + * for the Browse Engine, based on the brand of the provided DBMS. + * + * @param context the DSpace context + * @throws BrowseException if browse error + */ + public BrowseEngine(Context context) + throws BrowseException + { + // set the context + this.context = context; + + // prepare the data access object + dao = BrowseDAOFactory.getInstance(context); + } + + /** + * Perform a standard browse, which will return a BrowseInfo + * object that represents the results for the current page, the + * total number of results, the range, and information to construct + * previous and next links on any web page + * + * @param bs the scope of the browse + * @return the results of the browse + * @throws BrowseException if browse error + */ + public BrowseInfo browse(BrowserScope bs) + throws BrowseException + { + log.debug(LogManager.getHeader(context, "browse", "")); + + // first, load the browse scope into the object + this.scope = bs; + + // since we use it so much, get the browse index out of the + // scope and store as a member + browseIndex = scope.getBrowseIndex(); + + // now make the decision as to how to browse + if (browseIndex.isMetadataIndex() && !scope.isSecondLevel()) + { + // this is a single value browse type that has not gone to + // the second level (i.e. authors, not items by a given author) + return browseByValue(scope); + } + else + { + // this is the full browse type or a browse that has gone to + // the second level + return browseByItem(scope); + } + } + + /** + * Perform a limited browse, which only returns the results requested, + * without any extraneous information. To perform a full browse, use + * BrowseEngine.browse() above. This supports Item browse only, and does + * not currently support focus or values. This method is used, for example, + * to generate the Recently Submitted Items results. + * + * @param bs the scope of the browse + * @return the results of the browse + * @throws BrowseException if browse error + */ + public BrowseInfo browseMini(BrowserScope bs) + throws BrowseException + { + log.info(LogManager.getHeader(context, "browse_mini", "")); + + // load the scope into the object + this.scope = bs; + + // since we use it so much, get the browse index out of the + // scope and store as a member + browseIndex = scope.getBrowseIndex(); + + // get the table name that we are going to be getting our data from + dao.setTable(browseIndex.getTableName()); + + // tell the browse query whether we are ascending or descending on the value + dao.setAscending(scope.isAscending()); + + // define a clause for the WHERE clause which will allow us to constrain + // our browse to a specified community or collection + if (scope.inCollection() || scope.inCommunity()) + { + if (scope.inCollection()) + { + Collection col = (Collection) scope.getBrowseContainer(); + dao.setContainerTable("collection2item"); + dao.setContainerIDField("collection_id"); + dao.setContainerID(col.getID()); + } + else if (scope.inCommunity()) + { + Community com = (Community) scope.getBrowseContainer(); + dao.setContainerTable("communities2item"); + dao.setContainerIDField("community_id"); + dao.setContainerID(com.getID()); + } + } + + dao.setOffset(scope.getOffset()); + dao.setLimit(scope.getResultsPerPage()); + + // assemble the ORDER BY clause + String orderBy = browseIndex.getSortField(scope.isSecondLevel()); + if (scope.getSortBy() > 0) + { + orderBy = "sort_" + Integer.toString(scope.getSortBy()); + } + dao.setOrderField(orderBy); + + // now run the query + List results = dao.doQuery(); + + // construct the mostly empty BrowseInfo object to pass back + BrowseInfo browseInfo = new BrowseInfo(results, 0, scope.getResultsPerPage(), 0); + + // add the browse index to the Browse Info + browseInfo.setBrowseIndex(browseIndex); + + // set the sort option for the Browse Info + browseInfo.setSortOption(scope.getSortOption()); + + // tell the Browse Info which way we are sorting + browseInfo.setAscending(scope.isAscending()); + + // tell the browse info what the container for the browse was + if (scope.inCollection() || scope.inCommunity()) + { + browseInfo.setBrowseContainer(scope.getBrowseContainer()); + } + + browseInfo.setResultsPerPage(scope.getResultsPerPage()); + + browseInfo.setEtAl(scope.getEtAl()); + + return browseInfo; + } + + /** + * Browse the archive by the full item browse mechanism. This produces a + * BrowseInfo object which contains full BrowseItem objects as its result + * set. + * + * @param bs the scope of the browse + * @return the results of the browse + * @throws BrowseException if browse error + */ + private BrowseInfo browseByItem(BrowserScope bs) + throws BrowseException + { + log.info(LogManager.getHeader(context, "browse_by_item", "")); + try + { + // get the table name that we are going to be getting our data from + dao.setTable(browseIndex.getTableName()); + + // tell the browse query whether we are ascending or descending on the value + dao.setAscending(scope.isAscending()); + + // assemble the value clause + String rawValue = null; + if (scope.hasFilterValue() && scope.isSecondLevel()) + { + String value = scope.getFilterValue(); + rawValue = value; + + // make sure the incoming value is normalised + value = OrderFormat.makeSortString(value, scope.getFilterValueLang(), + scope.getBrowseIndex().getDataType()); + + dao.setAuthorityValue(scope.getAuthorityValue()); + + // set the values in the Browse Query + if (scope.isSecondLevel()) + { + dao.setFilterValueField("value"); + dao.setFilterValue(rawValue); + } + else + { + dao.setFilterValueField("sort_value"); + dao.setFilterValue(value); + } + dao.setFilterValuePartial(scope.getFilterValuePartial()); + + // to apply the filtering, we need the distinct and map tables for the index + dao.setFilterMappingTables(browseIndex.getDistinctTableName(), + browseIndex.getMapTableName()); + } + + // define a clause for the WHERE clause which will allow us to constrain + // our browse to a specified community or collection + if (scope.inCollection() || scope.inCommunity()) + { + if (scope.inCollection()) + { + Collection col = (Collection) scope.getBrowseContainer(); + dao.setContainerTable("collection2item"); + dao.setContainerIDField("collection_id"); + dao.setContainerID(col.getID()); + } + else if (scope.inCommunity()) + { + Community com = (Community) scope.getBrowseContainer(); + dao.setContainerTable("communities2item"); + dao.setContainerIDField("community_id"); + dao.setContainerID(com.getID()); + } + } + + // this is the total number of results in answer to the query + int total = getTotalResults(); + + // assemble the ORDER BY clause + String orderBy = browseIndex.getSortField(scope.isSecondLevel()); + if (scope.getSortBy() > 0) + { + orderBy = "sort_" + Integer.toString(scope.getSortBy()); + } + dao.setOrderField(orderBy); + + int offset = scope.getOffset(); + String rawFocusValue = null; + if (offset < 1 && (scope.hasJumpToItem() || scope.hasJumpToValue() || scope.hasStartsWith())) + { + // We need to convert these to an offset for the actual browse query. + // First, get a value that we can look up in the ordering field + rawFocusValue = getJumpToValue(); + + // make sure the incoming value is normalised + String focusValue = normalizeJumpToValue(rawFocusValue); + + log.debug("browsing using focus: " + focusValue); + + // Convert the focus value into an offset + offset = getOffsetForValue(focusValue); + } + + dao.setOffset(offset); + + // assemble the LIMIT clause + dao.setLimit(scope.getResultsPerPage()); + + // Holder for the results + List results = null; + + // Does this browse have any contents? + if (total > 0) + { + // now run the query + results = dao.doQuery(); + + // now, if we don't have any results, we are at the end of the browse. This will + // be because a starts_with value has been supplied for which we don't have + // any items. + if (results.size() == 0) + { + // In this case, we will calculate a new offset for the last page of results + offset = total - scope.getResultsPerPage(); + if (offset < 0) + { + offset = 0; + } + + // And rerun the query + dao.setOffset(offset); + results = dao.doQuery(); + } + } + else + { + // No records, so make an empty list + results = new ArrayList<>(); + } + + // construct the BrowseInfo object to pass back +// BrowseInfo browseInfo = new BrowseInfo(results, position, total, offset); + BrowseInfo browseInfo = new BrowseInfo(results, offset, total, offset); + + if (offset + scope.getResultsPerPage() < total) + { + browseInfo.setNextOffset(offset + scope.getResultsPerPage()); + } + + if (offset - scope.getResultsPerPage() > -1) + { + browseInfo.setPrevOffset(offset - scope.getResultsPerPage()); + } + + // add the browse index to the Browse Info + browseInfo.setBrowseIndex(browseIndex); + + // set the sort option for the Browse Info + browseInfo.setSortOption(scope.getSortOption()); + + // tell the Browse Info which way we are sorting + browseInfo.setAscending(scope.isAscending()); + + // tell the Browse Info which level of browse we are at + browseInfo.setBrowseLevel(scope.getBrowseLevel()); + + // set the browse value if there is one + browseInfo.setValue(rawValue); + + // set the browse authority key if there is one + browseInfo.setAuthority(scope.getAuthorityValue()); + + // set the focus value if there is one + browseInfo.setFocus(rawFocusValue); + + if (scope.hasJumpToItem()) + { + browseInfo.setFocusItem(scope.getJumpToItem()); + } + + // tell the browse info if it is working from a starts with parameter + browseInfo.setStartsWith(scope.hasStartsWith()); + + // tell the browse info what the container for the browse was + if (scope.inCollection() || scope.inCommunity()) + { + browseInfo.setBrowseContainer(scope.getBrowseContainer()); + } + + browseInfo.setResultsPerPage(scope.getResultsPerPage()); + + browseInfo.setEtAl(scope.getEtAl()); + + return browseInfo; + } + catch (SQLException e) + { + log.error("caught exception: ", e); + throw new BrowseException(e); + } + } + + /** + * Browse the archive by single values (such as the name of an author). This + * produces a BrowseInfo object that contains Strings as the results of + * the browse + * + * @param bs the scope of the browse + * @return the results of the browse + * @throws BrowseException if browse error + */ + private BrowseInfo browseByValue(BrowserScope bs) + throws BrowseException + { + log.info(LogManager.getHeader(context, "browse_by_value", "focus=" + bs.getJumpToValue())); + + try + { + // get the table name that we are going to be getting our data from + // this is the distinct table constrained to either community or collection + dao.setTable(browseIndex.getDistinctTableName()); + dao.setStartsWith("0".equals(scope.getStartsWith()) && !scope.getOrder().equals("ASC") ? "9" : normalizeJumpToValue(scope.getStartsWith())); + // remind the DAO that this is a distinct value browse, so it knows what sort + // of query to build + dao.setDistinct(true); + + // tell the browse query whether we are ascending or descending on the value + dao.setAscending(scope.isAscending()); + + // inform dao about the display frequencies flag + dao.setEnableBrowseFrequencies(browseIndex.isDisplayFrequencies()); + + // if we want to display frequencies, we need to pass the map table + if (browseIndex.isDisplayFrequencies()){ + dao.setFilterMappingTables(null, browseIndex.getMapTableName()); + } + + // set our constraints on community or collection + if (scope.inCollection() || scope.inCommunity()) + { + // Scoped browsing of distinct metadata requires the mapping + // table to be specified. + if (!browseIndex.isDisplayFrequencies()) + dao.setFilterMappingTables(null, browseIndex.getMapTableName()); + + if (scope.inCollection()) + { + Collection col = (Collection) scope.getBrowseContainer(); + dao.setContainerTable("collection2item"); + dao.setContainerIDField("collection_id"); + dao.setContainerID(col.getID()); + } + else if (scope.inCommunity()) + { + Community com = (Community) scope.getBrowseContainer(); + dao.setContainerTable("communities2item"); + dao.setContainerIDField("community_id"); + dao.setContainerID(com.getID()); + } + } + + // this is the total number of results in answer to the query + int total = getTotalResults(true); + + // set the ordering field (there is only one option) + dao.setOrderField("sort_value"); + + // assemble the focus clause if we are to have one + // it will look like one of the following + // - sort_value < myvalue + // = sort_1 > myvalue + dao.setJumpToField("sort_value"); + int offset = scope.getOffset(); + int limit = scope.getResultsPerPage(); + List results = null; + + String rawFocusValue = null; + if (offset < 1 && scope.hasJumpToValue() || scope.hasStartsWith()) { + // store the value to tell the Browse Info object which value we are browsing on + rawFocusValue = getJumpToValue(); + } + if ("0".equals(scope.getStartsWith())) { + int currentSW = scope.getOrder().equals("ASC") ? 0 : 9; + results = new ArrayList(); + // While we haven't reached results or the end + while (results.size() < scope.getResultsPerPage() && currentSW <= 9 && currentSW >= 0) { + List thisNumResults = dao.doValueQuery(); + // If set contains our results + if (offset < thisNumResults.size()) { + // If we have found the rest, add and exit loop + if (offset + limit < thisNumResults.size()) { + results.addAll(thisNumResults.subList(offset, offset + limit)); + break; + } else { // Else add all we can and query again + thisNumResults = thisNumResults.subList(offset, thisNumResults.size()); + results.addAll(thisNumResults); + offset = 0; + limit -= thisNumResults.size(); + } + } else { + offset -= thisNumResults.size(); + } + dao.setStartsWith(String.valueOf(scope.getOrder().equals("ASC") ? ++currentSW : --currentSW)); + } + + offset = scope.getOffset(); + } else { + // assemble the offset and limit + dao.setOffset(offset); + dao.setLimit(limit); + + // Does this browse have any contents? + if (total > 0) { + // now run the query + results = dao.doValueQuery(); + + // now, if we don't have any results, we are at the end of the browse. This will + // be because a starts_with value has been supplied for which we don't have + // any items. + if (results.size() == 0) { + // In this case, we will calculate a new offset for the last page of results + offset = total - scope.getResultsPerPage(); + if (offset < 0) { + offset = 0; + } + + // And rerun the query + dao.setOffset(offset); + results = dao.doValueQuery(); + } + } else { + // No records, so make an empty list + results = new ArrayList(); + } + } + + // construct the BrowseInfo object to pass back + BrowseInfo browseInfo = new BrowseInfo(results, offset, total, offset); + + if (offset + scope.getResultsPerPage() < total) + { + browseInfo.setNextOffset(offset + scope.getResultsPerPage()); + } + + if (offset - scope.getResultsPerPage() > -1) + { + browseInfo.setPrevOffset(offset - scope.getResultsPerPage()); + } + + // add the browse index to the Browse Info + browseInfo.setBrowseIndex(browseIndex); + + // set the sort option for the Browse Info + browseInfo.setSortOption(scope.getSortOption()); + + // tell the Browse Info which way we are sorting + browseInfo.setAscending(scope.isAscending()); + + // tell the Browse Info which level of browse we are at + browseInfo.setBrowseLevel(scope.getBrowseLevel()); + + // set the browse value if there is one + browseInfo.setFocus(rawFocusValue); + + // tell the browse info if it is working from a starts with parameter + browseInfo.setStartsWith(scope.hasStartsWith()); + + // tell the browse info what the container for the browse was + if (scope.inCollection() || scope.inCommunity()) + { + browseInfo.setBrowseContainer(scope.getBrowseContainer()); + } + + browseInfo.setResultsPerPage(scope.getResultsPerPage()); + + return browseInfo; + } + catch (SQLException e) + { + log.error("caught exception: ", e); + throw new BrowseException(e); + } + } + + /** + * Return the focus value. + * + * @return the focus value to use + * @throws BrowseException if browse error + */ + private String getJumpToValue() + throws BrowseException + { + log.debug(LogManager.getHeader(context, "get_focus_value", "")); + + // if the focus is by value, just return it + if (scope.hasJumpToValue()) + { + log.debug(LogManager.getHeader(context, "get_focus_value_return", "return=" + scope.getJumpToValue())); + return scope.getJumpToValue(); + } + + // if the focus is to start with, then we need to return the value of the starts with + if (scope.hasStartsWith()) + { + log.debug(LogManager.getHeader(context, "get_focus_value_return", "return=" + scope.getStartsWith())); + return scope.getStartsWith(); + } + + // since the focus is not by value, we need to obtain it + + // get the id of the item to focus on + int id = scope.getJumpToItem(); + + // get the table name. We don't really need to care about whether we are in a + // community or collection at this point. This is only for full or second + // level browse, so there is no need to worry about distinct value browsing + String tableName = browseIndex.getTableName(); + + // we need to make sure that we select from the correct column. If the sort option + // is the 0th option then we use sort_value, but if it is one of the others we have + // to select from that column instead. Otherwise, we end up missing the focus value + // to do comparisons in other columns. The use of the focus value needs to be consistent + // across the browse + SortOption so = scope.getSortOption(); + if (so == null || so.getNumber() == 0) + { + if (browseIndex.getSortOption() != null) + { + so = browseIndex.getSortOption(); + } + } + + String col = "sort_1"; + if (so.getNumber() > 0) + { + col = "sort_" + Integer.toString(so.getNumber()); + } + + + // now get the DAO to do the query for us, returning the highest + // string value in the given column in the given table for the + // item (I think) + String max = dao.doMaxQuery(col, tableName, id); + + log.debug(LogManager.getHeader(context, "get_focus_value_return", "return=" + max)); + + return max; + } + + /** + * Convert the value into an offset into the table for this browse + * + * @param value value + * @return the focus value to use + * @throws BrowseException if browse error + */ + private int getOffsetForValue(String value) + throws BrowseException + { + // we need to make sure that we select from the correct column. If the sort option + // is the 0th option then we use sort_value, but if it is one of the others we have + // to select from that column instead. Otherwise, we end up missing the focus value + // to do comparisons in other columns. The use of the focus value needs to be consistent + // across the browse + SortOption so = scope.getSortOption(); + if (so == null || so.getNumber() == 0) + { + if (browseIndex.getSortOption() != null) + { + so = browseIndex.getSortOption(); + } + } + + String col = "sort_1"; + if (so.getNumber() > 0) + { + col = "sort_" + Integer.toString(so.getNumber()); + } + + // now get the DAO to do the query for us, returning the highest + // string value in the given column in the given table for the + // item (I think) + return dao.doOffsetQuery(col, value, scope.isAscending()); + } + + /** + * Convert the value into an offset into the table for this browse + * + * @param value value + * @return the focus value to use + * @throws BrowseException if browse error + */ + private int getOffsetForDistinctValue(String value) + throws BrowseException + { + if (!browseIndex.isMetadataIndex()) + { + throw new IllegalArgumentException("getOffsetForDistinctValue called when not a metadata index"); + } + + // now get the DAO to do the query for us, returning the highest + // string value in the given column in the given table for the + // item (I think) + return dao.doDistinctOffsetQuery("sort_value", value, scope.isAscending()); + } + + /** + * Return a normalized focus value. If there is no normalization that can be performed, + * return the focus value that is passed in. + * + * @param value a focus value to normalize + * @return the normalized focus value + * @throws BrowseException if browse error + */ + private String normalizeJumpToValue(String value) + throws BrowseException + { + // If the scope has a focus value (focus by value) + if (scope.hasJumpToValue()) + { + // Normalize it based on the specified language as appropriate for this index + return OrderFormat.makeSortString(scope.getJumpToValue(), scope.getJumpToValueLang(), scope.getBrowseIndex().getDataType()); + } + else if (scope.hasStartsWith()) + { + // Scope has a starts with, so normalize that instead + return OrderFormat.makeSortString(scope.getStartsWith(), null, scope.getBrowseIndex().getDataType()); + } + + // No focus value on the scope (ie. focus by id), so just return the passed focus value + // This is useful in cases where we have pulled a focus value from the index + // which will already be normalized, and avoids another DB lookup + return value; + } + + /** + * Get the total number of results for the browse. This is the same as + * calling getTotalResults(false) + * + * @return total + * @throws SQLException if database error + * @throws BrowseException if browse error + */ + private int getTotalResults() + throws SQLException, BrowseException + { + return getTotalResults(false); + } + + /** + * Get the total number of results. The argument determines whether this is a distinct + * browse or not as this has an impact on how results are counted + * + * @param distinct is this a distinct browse or not + * @return the total number of results available in this type of browse + * @throws SQLException if database error + * @throws BrowseException if browse error + */ + private int getTotalResults(boolean distinct) + throws SQLException, BrowseException + { + log.debug(LogManager.getHeader(context, "get_total_results", "distinct=" + distinct)); + + // tell the browse query whether we are distinct + dao.setDistinct(distinct); + + // ensure that the select is set to "*" + String[] select = { "*" }; + dao.setCountValues(select); + + // FIXME: it would be nice to have a good way of doing this in the DAO + // now reset all of the fields that we don't want to have constraining + // our count, storing them locally to reinstate later + String focusField = dao.getJumpToField(); + String focusValue = dao.getJumpToValue(); + String orderField = dao.getOrderField(); + int limit = dao.getLimit(); + int offset = dao.getOffset(); + + dao.setJumpToField(null); + dao.setJumpToValue(null); + dao.setOrderField(null); + dao.setLimit(-1); + dao.setOffset(-1); + + // perform the query and get the result + int count = dao.doCountQuery(); + if ("0".equals(dao.getStartsWith())) { + for (int x = 1; x <= 9; x++) { + dao.setStartsWith(String.valueOf(x)); + count += dao.doCountQuery(); + } + dao.setStartsWith("0"); + } else if ("9".equals(dao.getStartsWith())) { + for (int x = 8; x >= 0; x--) { + dao.setStartsWith(String.valueOf(x)); + count += dao.doCountQuery(); + } + dao.setStartsWith("9"); + } + + // now put back the values we removed for this method + dao.setJumpToField(focusField); + dao.setJumpToValue(focusValue); + dao.setOrderField(orderField); + dao.setLimit(limit); + dao.setOffset(offset); + dao.setCountValues(null); + + log.debug(LogManager.getHeader(context, "get_total_results_return", "return=" + count)); + + return count; + } +} \ No newline at end of file diff --git a/dspace/modules/additions/src/main/java/org/dspace/browse/BrowseInfo.java b/dspace/modules/additions/src/main/java/org/dspace/browse/BrowseInfo.java new file mode 100644 index 0000000000..88afe7d6cd --- /dev/null +++ b/dspace/modules/additions/src/main/java/org/dspace/browse/BrowseInfo.java @@ -0,0 +1,927 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.browse; + +import java.sql.SQLException; +import java.util.Collections; +import java.util.Iterator; +import java.util.List; + +import org.dspace.content.*; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.ItemService; +import org.dspace.sort.SortOption; + +/** + * The results of a Browse, including all the contextual information about + * the query, as well as the results and associated information to create + * pageable navigation. + * + * @author Richard Jones + */ +public class BrowseInfo +{ + /** + * The results of the browse. + * FIXME: Unable to generify due to mixed usage + */ + private List results; + + /** + * The position of the first element of results within the Browse index. + * Positions begin with 0. + */ + private int overallPosition; + + /** + * The position of the requested object within the results. Offsets begin + * with 0. + */ + private int offset; + + /** + * The total number of items in the browse index. + */ + private int total; + + /** + * True if this browse was cached. + */ + private boolean cached; + + /** the browse index to which this pertains */ + private BrowseIndex browseIndex; + + /** the sort option being used */ + private SortOption sortOption; + + /** is the browse ascending or descending */ + private boolean ascending; + + /** what level of browse are we in? full and single front pages are 0, single value browse is 1 */ + private int level = 0; + + /** the value browsed upon */ + private String value; + + /** the authority key browsed upon */ + private String authority; + + /** is this a "starts_with" browse? */ + private boolean startsWith = false; + + /** Collection we are constrained to */ + private Collection collection; + + /** Community we are constrained to */ + private Community community; + + /** offset of the item at the top of the next page */ + private int nextOffset = -1; + + /** offset of the item at the top of the previous page */ + private int prevOffset = -1; + + /** the value upon which we are focusing */ + private String focus; + + /** number of results to display per page */ + private int resultsPerPage = -1; + + /** database id of the item upon which we are focusing */ + private int focusItem = -1; + + /** number of metadata elements to display before truncating using "et al" */ + private int etAl = -1; + + protected ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + + /** + * Constructor + * FIXME: Unable to generify due to mixed usage + * + * @param results + * A List of Browse results + * @param overallPosition + * The position of the first returned item in the overall index + * @param total + * The total number of items in the index + * @param offset + * The position of the requested item in the set of results + */ + public BrowseInfo(List results, int overallPosition, int total, int offset) + { + if (results == null) + { + throw new IllegalArgumentException("Null result list not allowed"); + } + + this.results = Collections.unmodifiableList(results); + this.overallPosition = overallPosition; + this.total = total; + this.offset = offset; + } + + /** + * @return the number of metadata fields at which to truncate with "et al" + */ + public int getEtAl() + { + return etAl; + } + + /** + * set the number of metadata fields at which to truncate with "et al" + * + * @param etAl + */ + public void setEtAl(int etAl) + { + this.etAl = etAl; + } + + /** + * @return Returns the focusItem. + */ + public int getFocusItem() + { + return focusItem; + } + + /** + * @param focusItem The focusItem to set. + */ + public void setFocusItem(int focusItem) + { + this.focusItem = focusItem; + } + + /** + * Does this browse have an item focus (as opposed to one of: no focus, + * a value focus) + * + * @return true if item focus, false if not + */ + public boolean hasItemFocus() + { + if (focusItem == -1) + { + return false; + } + return true; + } + + /** + * @return Returns the resultsPerPage. + */ + public int getResultsPerPage() + { + return resultsPerPage; + } + + /** + * @param resultsPerPage The resultsPerPage to set. + */ + public void setResultsPerPage(int resultsPerPage) + { + this.resultsPerPage = resultsPerPage; + } + + /** + * Is there a value associated with this browse + * + * @return true if a value, false if not + */ + public boolean hasValue() + { + if (this.value != null) + { + return true; + } + return false; + } + + /** + * Is there an authority key associated with this browse + * + * @return true if an authority key, false if not + */ + public boolean hasAuthority() + { + if (this.authority != null) + { + return true; + } + return false; + } + + /** + * Are there results for this browse, or was the result set empty? + * + * @return true if results, false if not + */ + public boolean hasResults() + { + if (results.size() > 0) + { + return true; + } + return false; + } + + /** + * @param focus the value to focus the browse around + */ + public void setFocus(String focus) + { + this.focus = focus; + } + + /** + * @return the value to focus the browse around + */ + public String getFocus() + { + return this.focus; + } + + /** + * Set the DSpaceObject that is the container for this browse. If this + * is not of type Collection or Community, this method will throw an + * exception + * + * @param dso the container object; a Community or Collection + * @throws BrowseException if browse error + */ + public void setBrowseContainer(DSpaceObject dso) + throws BrowseException + { + if (dso instanceof Collection) + { + this.collection = (Collection) dso; + } + else if (dso instanceof Community) + { + this.community = (Community) dso; + } + else + { + throw new BrowseException("The container must be a community or a collection"); + } + } + + /** + * Obtain a DSpaceObject that represents the container object. This will be + * a Community or a Collection + * + * @return A DSpaceObject representing a Community or a Collection + */ + public DSpaceObject getBrowseContainer() + { + if (this.collection != null) + { + return this.collection; + } + if (this.community != null) + { + return this.community; + } + return null; + } + + /** + * @param level the browse level + */ + public void setBrowseLevel(int level) + { + this.level = level; + } + + /** + * @return the browse level + */ + public int getBrowseLevel() + { + return this.level; + } + + /** + * @param offset the database id of the item at the top of the next page + */ + public void setNextOffset(int offset) + { + this.nextOffset = offset; + } + + /** + * @return the database id of the item at the top of the next page + */ + public int getNextOffset() + { + return this.nextOffset; + } + + /** + * @return Returns the ascending. + */ + public boolean isAscending() + { + return ascending; + } + + /** + * @param ascending The ascending to set. + */ + public void setAscending(boolean ascending) + { + this.ascending = ascending; + } + + /** + * @return Returns the browseIndex. + */ + public BrowseIndex getBrowseIndex() + { + return browseIndex; + } + + /** + * @param browseIndex The browseIndex to set. + */ + public void setBrowseIndex(BrowseIndex browseIndex) + { + this.browseIndex = browseIndex; + } + + /** + * @return Returns the prevItem. + */ + public int getPrevOffset() + { + return prevOffset > -1 ? prevOffset : 0; + } + + /** + * @param prevOffset The prevOffset to set. + */ + public void setPrevOffset(int prevOffset) + { + this.prevOffset = prevOffset; + } + + /** + * @return Returns the sortOption. + */ + public SortOption getSortOption() + { + return sortOption; + } + + /** + * @param sortOption The sortOption to set. + */ + public void setSortOption(SortOption sortOption) + { + this.sortOption = sortOption; + } + + /** + * @return Returns the startsWith. + */ + public boolean isStartsWith() + { + return startsWith; + } + + /** + * @param startsWith The startsWith to set. + */ + public void setStartsWith(boolean startsWith) + { + this.startsWith = startsWith; + } + + /** + * @return Returns the value. + */ + public String getValue() + { + return value; + } + + /** + * @param value The value to set. + */ + public void setValue(String value) + { + this.value = value; + } + + /** + * @return Returns the authority key. + */ + public String getAuthority() + { + return authority; + } + + /** + * @param authority The authority key to set. + */ + public void setAuthority(String authority) + { + this.authority = authority; + } + + /** + * is this a top level (0) browse? Examples of this are a full item + * browse or a single browse. Other browse types are considered + * second level (1) + * + * @return true if top level, false if not + */ + public boolean isTopLevel() + { + if (this.level == 0) + { + return true; + } + return false; + } + + /** + * Is this a second level (1) browse? Examples of this are a single + * value browse (e.g. all items by a given author) + * + * @return true if second level, false if not + */ + public boolean isSecondLevel() + { + if (this.level == 1) + { + return true; + } + return false; + } + + /** + * The results of the Browse. Each member of the list is either a String array + * (for the authors browse: first element the value, second element the authority key) + * or an {@link org.dspace.content.Item}(for the + * other browses). + * + * @return Result list. This list cannot be modified. + */ + public List getResults() + { + return results; + } + + public void setResults(List results) { + this.results = results; + } + + /** + * Return the results of the Browse as an array of String array. + * The first element (i.e. index 0) is the value, the second is the authority key + * + * @return The results of the Browse as a String array. + */ + public String[][] getStringResults() + { + return (String[][]) results.toArray(new String[results.size()][2]); + } + + /** + * @deprecated + * @return an empty array of Item. + */ + public Item[] getItemResults() + { + return new Item[0]; + } + + /** + * Return the results of the Browse as a BrowseItem array + * + * @return the results of the browse as a BrowseItem array + */ + public List getBrowseItemResults() + { + return results; + } + + /** + * Return the number of results. + * + * @return The number of results. + */ + public int getResultCount() + { + return results.size(); + } + + /** + * Return the position of the results in index being browsed. This is 0 for + * the start of the index. + * + * @return The position of the results in index being browsed. + */ + public int getOverallPosition() + { + return overallPosition; + } + + /** + * Return the total number of items in the index. + * + * @return The total number of items in the index. + */ + public int getTotal() + { + return total; + } + + public void setTotal(int total) { + this.total = total; + } + + /** + * Return the position of the requested item or value in the set of results. + * + * @return The position of the requested item or value in the set of results + */ + public int getOffset() + { + return offset; + } + + /** + * True if there are no previous results from the browse. + * + * @return True if there are no previous results from the browse + */ + public boolean isFirst() + { + return overallPosition == 0; + } + + /** + * True if these are the last results from the browse. + * + * @return True if these are the last results from the browse + */ + public boolean isLast() + { + return (overallPosition + getResultCount()) == total; + } + + /** + * True if this browse was cached. + * @return true/false + */ + public boolean wasCached() + { + return cached; + } + + /** + * Set whether this browse was cached. + */ + void setCached(boolean cached) + { + this.cached = cached; + } + + /** + * are we browsing within a Community container? + * + * @return true if in community, false if not + */ + public boolean inCommunity() + { + if (this.community != null) + { + return true; + } + return false; + } + + /** + * are we browsing within a Collection container + * + * @return true if in collection, false if not + */ + public boolean inCollection() + { + if (this.collection != null) + { + return true; + } + return false; + } + + /** + * Are there further results for the browse that haven't been returned yet? + * + * @return true if next page, false if not + */ + public boolean hasNextPage() + { + if (nextOffset > -1) + { + return true; + } + return false; + } + + /** + * Are there results prior to these that haven't been returned here? + * + * @return true if previous page, false if not + */ + public boolean hasPrevPage() + { + if (offset > 0) + { + return true; + } + return false; + } + + /** + * Does this browse have a focus? + * + * @return true if focus, false if not + */ + public boolean hasFocus() + { + if ("".equals(focus) || focus == null) + { + return false; + } + return true; + } + + /** + * Get an integer representing the number within the total set of results which + * marks the position of the first result in the current sub-set + * + * @return the start point of the browse page + */ + public int getStart() + { + return overallPosition + 1; + } + + /** + * Get an integer representing the number within the total set of results which + * marks the position of the last result in the current sub-set + * + * @return the end point of the browse page + */ + public int getFinish() + { + return overallPosition + results.size(); + } + + /** + * Utility method for obtaining a string representation of the browse. This is + * useful only for debug + * @return String representation + */ + public String toString() + { + try + { + StringBuffer sb = new StringBuffer(); + + // calculate the range for display + String from = Integer.toString(overallPosition + 1); + String to = Integer.toString(overallPosition + results.size()); + String of = Integer.toString(total); + + // report on the positional information of the browse + sb.append("BrowseInfo String Representation: "); + sb.append("Browsing " + from + " to " + to + " of " + of + " "); + + // insert the information about which index + sb.append("in index: " + browseIndex.getName() + + " (data type: " + browseIndex.getDataType() + + ", display type: " + browseIndex.getDisplayType() + ") "); + + sb.append("||"); + + // report on the browse scope container + String container = "all of DSpace"; + DSpaceObject theContainer = null; + if (inCollection()) + { + container = "collection"; + theContainer = this.collection; + } + else if (inCommunity()) + { + container = "community"; + theContainer = this.community; + } + + String containerID = "no id available/necessary"; + if (theContainer != null) + { + containerID = theContainer.getID().toString() + " (" + theContainer.getHandle() + ")"; + } + + sb.append("Browsing in " + container + ": " + containerID); + sb.append("||"); + + // load the item list display configuration + ItemListConfig config = new ItemListConfig(); + + // some information about the columns to be displayed + if (browseIndex.isItemIndex()) + { + sb.append("Listing over " + Integer.toString(config.numCols()) + " columns: "); + for (int k = 1; k <= config.numCols(); k++) + { + if (k > 1) + { + sb.append(","); + } + String[] meta = config.getMetadata(k); + sb.append(meta[0] + "." + meta[1] + "." + meta[2]); + } + + if (value != null) + { + sb.append(" on value: ").append(value); + } + + if (isStartsWith()) + { + sb.append(" sort column starting with: ").append(focus); + } + else if (hasFocus()) + { + sb.append(" sort column focus: ").append(focus); + } + } + else if (browseIndex.isMetadataIndex()) + { + sb.append("Listing single column: ").append(browseIndex.getMetadata()); + if (isStartsWith()) + { + sb.append(" sort column starting with: ").append(focus); + } + else if (hasFocus()) + { + sb.append(" sort column focus: ").append(focus); + } + } + + sb.append("||"); + + // some information about how the data is sorted + String direction = (ascending ? "ASC" : "DESC"); + sb.append("Sorting by: " + sortOption.getMetadata() + " " + direction + + " (option " + Integer.toString(sortOption.getNumber()) + ")"); + sb.append("||"); + + // output the results + if (browseIndex.isMetadataIndex() && !isSecondLevel()) + { + sb.append(valueListingString()); + } + else if (browseIndex.isItemIndex() || isSecondLevel()) + { + sb.append(fullListingString(config)); + } + + sb.append("||"); + + // tell us what the next and previous values are going to be + sb.append("Top of next page: "); + if (hasNextPage()) + { + sb.append("offset: ").append(Integer.toString(this.nextOffset)); + } + else + { + sb.append("n/a"); + } + sb.append(";"); + + sb.append("Top of previous page: "); + if (hasPrevPage()) + { + sb.append("offset: ").append(Integer.toString(this.prevOffset)); + } + else + { + sb.append("n/a"); + } + + sb.append("||"); + + return sb.toString(); + } + catch (SQLException e) + { + return e.getMessage(); + } + catch (BrowseException e) + { + return e.getMessage(); + } + } + + /** + * A utility method for generating a string to represent a single item's + * entry in the browse + * + * @param config + * @return + * @throws SQLException if database error + */ + private String fullListingString(ItemListConfig config) + throws SQLException + { + // report on all the results contained herein + StringBuffer sb = new StringBuffer(); + + Iterator itr = results.iterator(); + while (itr.hasNext()) + { + Item bi = (Item) itr.next(); + if (bi == null) + { + sb.append("{{ NULL ITEM }}"); + break; + } + sb.append("{{Item ID: " + bi.getID().toString() + " :: "); + + for (int j = 1; j <= config.numCols(); j++) + { + String[] md = config.getMetadata(j); + if (md == null) + { + sb.append("{{ NULL METADATA }}"); + break; + } + List values = itemService.getMetadata(bi, md[0], md[1], md[2], Item.ANY); + StringBuffer value = new StringBuffer(); + if (values != null) + { + for (int i = 0; i < values.size(); i++) + { + if (i > 0) + { + value.append(","); + } + value.append(values.get(i).getValue()); + } + } + else + { + value.append("-"); + } + String metadata = "[" + md[0] + "." + md[1] + "." + md[2] + ":" + value.toString() + "]"; + sb.append(metadata); + } + + sb.append("}}"); + } + + return sb.toString(); + } + + /** + * A utility method for representing a single value in the browse + * + * @return + */ + private String valueListingString() + { + // report on all the results contained herein + StringBuffer sb = new StringBuffer(); + + Iterator itr = results.iterator(); + while (itr.hasNext()) + { + String theValue = (String) itr.next(); + if (theValue == null) + { + sb.append("{{ NULL VALUE }}"); + break; + } + sb.append("{{Value: " + theValue + "}}"); + } + + return sb.toString(); + } +} \ No newline at end of file diff --git a/dspace/modules/additions/src/main/java/org/dspace/browse/SolrBrowseDAO.java b/dspace/modules/additions/src/main/java/org/dspace/browse/SolrBrowseDAO.java new file mode 100644 index 0000000000..082770c8f6 --- /dev/null +++ b/dspace/modules/additions/src/main/java/org/dspace/browse/SolrBrowseDAO.java @@ -0,0 +1,837 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.browse; + +import org.apache.commons.lang3.StringUtils; +import org.apache.log4j.Logger; +import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; +import org.dspace.content.DSpaceObject; +import org.dspace.content.Item; +import org.dspace.core.Constants; +import org.dspace.core.Context; +import org.dspace.discovery.*; +import org.dspace.discovery.DiscoverQuery.SORT_ORDER; +import org.dspace.discovery.DiscoverResult.FacetResult; +import org.dspace.discovery.DiscoverResult.SearchDocument; +import org.dspace.discovery.configuration.DiscoveryConfigurationParameters; +import org.dspace.services.factory.DSpaceServicesFactory; + +import java.io.Serializable; +import java.sql.SQLException; +import java.util.*; + +/** + * + * @author Andrea Bollini (CILEA) + * @author Adán Román Ruiz at arvo.es (bugfix) + * @author Panagiotis Koutsourakis (National Documentation Centre) (bugfix) + * @author Kostas Stamatis (National Documentation Centre) (bugfix) + * + */ +public class SolrBrowseDAO implements BrowseDAO +{ + public SolrBrowseDAO(Context context) + { + this.context = context; + } + + static private class FacetValueComparator + implements Comparator, Serializable + { + @Override + public int compare(Object o1, Object o2) + { + String s1 = "", s2 = ""; + if (o1 instanceof FacetResult && o2 instanceof String) + { + FacetResult c = (FacetResult) o1; + s1 = c.getSortValue(); + s2 = (String) o2; + } + else if (o2 instanceof FacetResult && o1 instanceof String) + { + FacetResult c = (FacetResult) o2; + s1 = (String) o1; + s2 = c.getSortValue(); + } + // both object are FacetResult so they are already sorted + return s1.compareTo(s2); + } + } + + /** Log4j log */ + private static final Logger log = Logger.getLogger(SolrBrowseDAO.class); + + /** The DSpace context */ + private final Context context; + + // SQL query related attributes for this class + + /** table(s) to select from */ + private String table = null; + + /** field to look for focus value in */ + private String focusField = null; + + /** value to start browse from in focus field */ + private String focusValue = null; + + private String startsWith = null; + + /** field to look for value in */ + private String valueField = null; + + /** value to restrict browse to (e.g. author name) */ + private String value = null; + + private String authority = null; + + /** exact or partial matching of the value */ + private boolean valuePartial = false; + + /** the table that defines the mapping for the relevant container */ + private String containerTable = null; + + /** + * the name of the field which contains the container id (e.g. + * collection_id) + */ + private String containerIDField = null; + + /** the database id of the container we are constraining to */ + private UUID containerID = null; + + /** the column that we are sorting results by */ + private String orderField = null; + + /** whether to sort results ascending or descending */ + private boolean ascending = true; + + /** the limit of number of results to return */ + private int limit = -1; + + /** the offset of the start point */ + private int offset = 0; + + /** whether to use the equals comparator in value comparisons */ + private boolean equalsComparator = true; + + /** whether this is a distinct browse or not */ + private boolean distinct = false; + + private String facetField; + + protected AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); + + // administrative attributes for this class + + + SearchService searcher = DSpaceServicesFactory.getInstance().getServiceManager().getServiceByName( + SearchService.class.getName(), SearchService.class); + + private DiscoverResult sResponse = null; + + private boolean itemsWithdrawn = false; + private boolean itemsDiscoverable = true; + + private boolean showFrequencies; + + private DiscoverResult getSolrResponse() throws BrowseException + { + if (sResponse == null) + { + DiscoverQuery query = new DiscoverQuery(); + addLocationScopeFilter(query); + addStatusFilter(query); + if (distinct) + { + DiscoverFacetField dff; + if (StringUtils.isNotBlank(startsWith)) { + dff = new DiscoverFacetField(facetField, + DiscoveryConfigurationParameters.TYPE_TEXT, -1, + DiscoveryConfigurationParameters.SORT.VALUE, startsWith); + } else { + dff = new DiscoverFacetField(facetField, + DiscoveryConfigurationParameters.TYPE_TEXT, -1, + DiscoveryConfigurationParameters.SORT.VALUE); + } + query.addFacetField(dff); + query.setFacetMinCount(1); + query.setMaxResults(0); + } + else + { + query.setMaxResults(limit/* > 0 ? limit : 20*/); + if (offset > 0) + { + query.setStart(offset); + } + + // caution check first authority, value is always present! + if (authority != null) + { + query.addFilterQueries("{!field f="+facetField + "_authority_filter}" + + authority); + } + else if (value != null && !valuePartial) + { + query.addFilterQueries("{!field f="+facetField + "_value_filter}" + value); + } + else if (valuePartial) + { + query.addFilterQueries("{!field f="+facetField + "_partial}" + value); + } + // filter on item to be sure to don't include any other object + // indexed in the Discovery Search core + query.addFilterQueries("search.resourcetype:" + Constants.ITEM); + if (orderField != null) + { + query.setSortField("bi_" + orderField + "_sort", + ascending ? SORT_ORDER.asc : SORT_ORDER.desc); + } + } + try + { + sResponse = searcher.search(context, query, itemsWithdrawn + || !itemsDiscoverable); + } + catch (SearchServiceException e) + { + throw new BrowseException(e); + } + } + return sResponse; + } + + private void addStatusFilter(DiscoverQuery query) + { + if (itemsWithdrawn) + { + query.addFilterQueries("withdrawn:true"); + } + else if (!itemsDiscoverable) + { + query.addFilterQueries("discoverable:false"); + // TODO + + try + { + if (!authorizeService.isAdmin(context) + && (authorizeService.isCommunityAdmin(context) + || authorizeService.isCollectionAdmin(context))) + { + query.addFilterQueries(searcher.createLocationQueryForAdministrableItems(context)); + } + } + catch (SQLException ex) + { + log.error(ex); + } + } + } + + private void addLocationScopeFilter(DiscoverQuery query) + { + if (containerID != null) + { + if (containerIDField.startsWith("collection")) + { + query.addFilterQueries("location.coll:" + containerID); + } + else if (containerIDField.startsWith("community")) + { + query.addFilterQueries("location.comm:" + containerID); + } + } + } + + @Override + public int doCountQuery() throws BrowseException + { + DiscoverResult resp = getSolrResponse(); + int count = 0; + if (distinct) + { + List facetResults = resp.getFacetResult(facetField); + count = facetResults.size(); + } + else + { + // we need to cast to int to respect the BrowseDAO contract... + count = (int) resp.getTotalSearchResults(); + // FIXME null the response cache + // the BrowseEngine send fake argument to the BrowseDAO for the + // count... + sResponse = null; + } + return count; + } + + @Override + public List doValueQuery() throws BrowseException + { + DiscoverResult resp = getSolrResponse(); + List facet = resp.getFacetResult(facetField); + int count = doCountQuery(); + int start = offset > 0 ? offset : 0; + int max = limit > 0 ? limit : count; //if negative, return everything + List result = new ArrayList<>(); + if (ascending) + { + for (int i = start; i < (start + max) && i < count; i++) + { + FacetResult c = facet.get(i); + String freq = showFrequencies ? String.valueOf(c.getCount()) + : ""; + result.add(new String[] { c.getDisplayedValue(), + c.getAuthorityKey(), freq }); + } + } + else + { + for (int i = count - start - 1; i >= count - (start + max) + && i >= 0; i--) + { + FacetResult c = facet.get(i); + String freq = showFrequencies ? String.valueOf(c.getCount()) + : ""; + result.add(new String[] { c.getDisplayedValue(), + c.getAuthorityKey(), freq }); + } + } + + return result; + } + + @Override + public List doQuery() throws BrowseException + { + DiscoverResult resp = getSolrResponse(); + + List bitems = new ArrayList<>(); + for (DSpaceObject solrDoc : resp.getDspaceObjects()) + { + // FIXME introduce project, don't retrieve Item immediately when + // processing the query... + Item item = (Item) solrDoc; + bitems.add(item); + } + return bitems; + } + + @Override + public String doMaxQuery(String column, String table, int itemID) + throws BrowseException + { + DiscoverQuery query = new DiscoverQuery(); + query.setQuery("search.resourceid:" + itemID + + " AND search.resourcetype:" + Constants.ITEM); + query.setMaxResults(1); + DiscoverResult resp = null; + try + { + resp = searcher.search(context, query); + } + catch (SearchServiceException e) + { + throw new BrowseException(e); + } + if (resp.getTotalSearchResults() > 0) + { + SearchDocument doc = resp.getSearchDocument( + resp.getDspaceObjects().get(0)).get(0); + return (String) doc.getSearchFieldValues(column).get(0); + } + return null; + } + + @Override + public int doOffsetQuery(String column, String value, boolean isAscending) + throws BrowseException + { + DiscoverQuery query = new DiscoverQuery(); + addLocationScopeFilter(query); + addStatusFilter(query); + query.setMaxResults(0); + query.addFilterQueries("search.resourcetype:" + Constants.ITEM); + + // We need to take into account the fact that we may be in a subset of the items + if (authority != null) + { + query.addFilterQueries("{!field f="+facetField + "_authority_filter}" + + authority); + } + else if (this.value != null && !valuePartial) + { + query.addFilterQueries("{!field f="+facetField + "_value_filter}" + this.value); + } + else if (valuePartial) + { + query.addFilterQueries("{!field f="+facetField + "_partial}" + this.value); + } + + if (isAscending) + { + query.setQuery("bi_"+column + "_sort" + ": [* TO \"" + value + "\"}"); + } + else + { + query.setQuery("bi_" + column + "_sort" + ": {\"" + value + "\" TO *]"); + query.addFilterQueries("-(bi_" + column + "_sort" + ":" + value + "*)"); + } + boolean includeUnDiscoverable = itemsWithdrawn || !itemsDiscoverable; + DiscoverResult resp = null; + try + { + resp = searcher.search(context, query, includeUnDiscoverable); + } + catch (SearchServiceException e) + { + throw new BrowseException(e); + } + return (int) resp.getTotalSearchResults(); + } + + @Override + public int doDistinctOffsetQuery(String column, String value, + boolean isAscending) throws BrowseException + { + DiscoverResult resp = getSolrResponse(); + List facets = resp.getFacetResult(facetField); + Comparator comparator = new SolrBrowseDAO.FacetValueComparator(); + Collections.sort(facets, comparator); + int x = Collections.binarySearch(facets, value, comparator); + int ascValue = (x >= 0) ? x : -(x + 1); + if (isAscending) + { + return ascValue; + } + else + { + return doCountQuery() - ascValue; + } + } + + @Override + public boolean isEnableBrowseFrequencies() + { + return showFrequencies; + } + + @Override + public void setEnableBrowseFrequencies(boolean enableBrowseFrequencies) + { + showFrequencies = enableBrowseFrequencies; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getContainerID() + */ + @Override + public UUID getContainerID() + { + return containerID; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getContainerIDField() + */ + @Override + public String getContainerIDField() + { + return containerIDField; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getContainerTable() + */ + @Override + public String getContainerTable() + { + return containerTable; + } + + // FIXME is this in use? + @Override + public String[] getCountValues() + { + return null; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getFocusField() + */ + @Override + public String getJumpToField() + { + return focusField; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getFocusValue() + */ + @Override + public String getJumpToValue() + { + return focusValue; + } + + @Override + public void setStartsWith(String startsWith) { + sResponse = null; + this.startsWith = startsWith; + } + + @Override + public String getStartsWith() { + return startsWith; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getLimit() + */ + @Override + public int getLimit() + { + return limit; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getOffset() + */ + @Override + public int getOffset() + { + return offset; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getOrderField() + */ + @Override + public String getOrderField() + { + return orderField; + } + + // is this in use? + @Override + public String[] getSelectValues() + { + return null; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getTable() + */ + @Override + public String getTable() + { + return table; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getValue() + */ + @Override + public String getFilterValue() + { + return value; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#getValueField() + */ + @Override + public String getFilterValueField() + { + return valueField; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#isAscending() + */ + @Override + public boolean isAscending() + { + return ascending; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#isDistinct() + */ + @Override + public boolean isDistinct() + { + return this.distinct; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setAscending(boolean) + */ + @Override + public void setAscending(boolean ascending) + { + this.ascending = ascending; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setContainerID(int) + */ + @Override + public void setContainerID(UUID containerID) + { + this.containerID = containerID; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setContainerIDField(java.lang.String) + */ + @Override + public void setContainerIDField(String containerIDField) + { + this.containerIDField = containerIDField; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setContainerTable(java.lang.String) + */ + @Override + public void setContainerTable(String containerTable) + { + this.containerTable = containerTable; + + } + + // is this in use? + @Override + public void setCountValues(String[] fields) + { + // this.countValues = fields; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setDistinct(boolean) + */ + @Override + public void setDistinct(boolean bool) + { + this.distinct = bool; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setEqualsComparator(boolean) + */ + @Override + public void setEqualsComparator(boolean equalsComparator) + { + this.equalsComparator = equalsComparator; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setFocusField(java.lang.String) + */ + @Override + public void setJumpToField(String focusField) + { + this.focusField = focusField; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setFocusValue(java.lang.String) + */ + @Override + public void setJumpToValue(String focusValue) + { + this.focusValue = focusValue; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setLimit(int) + */ + @Override + public void setLimit(int limit) + { + this.limit = limit; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setOffset(int) + */ + @Override + public void setOffset(int offset) + { + this.offset = offset; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setOrderField(java.lang.String) + */ + @Override + public void setOrderField(String orderField) + { + this.orderField = orderField; + + } + + // is this in use? + @Override + public void setSelectValues(String[] selectValues) + { + // this.selectValues = selectValues; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setTable(java.lang.String) + */ + @Override + public void setTable(String table) + { + if (table.equals(BrowseIndex.getWithdrawnBrowseIndex().getTableName())) + { + itemsWithdrawn = true; + } + else if (table.equals(BrowseIndex.getPrivateBrowseIndex().getTableName())) + { + itemsDiscoverable = false; + } + facetField = table; + } + + @Override + public void setFilterMappingTables(String tableDis, String tableMap) + { + if (tableDis != null) + { + this.facetField = tableDis; + } + // this.fields = tableDis; + // this.tableMap = tableMap; + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setValue(java.lang.String) + */ + @Override + public void setFilterValue(String value) + { + this.value = value; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setFilterValuePartial(boolean) + */ + @Override + public void setFilterValuePartial(boolean part) + { + this.valuePartial = part; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#setValueField(java.lang.String) + */ + @Override + public void setFilterValueField(String valueField) + { + this.valueField = valueField; + + } + + /* + * (non-Javadoc) + * + * @see org.dspace.browse.BrowseDAO#useEqualsComparator() + */ + @Override + public boolean useEqualsComparator() + { + return equalsComparator; + } + + @Override + public String getAuthorityValue() + { + return authority; + } + + @Override + public void setAuthorityValue(String value) + { + this.authority = value; + } +} diff --git a/dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatAuthor.java b/dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatAuthor.java new file mode 100644 index 0000000000..0d41a3fb53 --- /dev/null +++ b/dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatAuthor.java @@ -0,0 +1,28 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.sort; + +import org.dspace.text.filter.DecomposeDiactritics; +import org.dspace.text.filter.LowerCaseAndTrim; +import org.dspace.text.filter.StripDiacritics; +import org.dspace.text.filter.TextFilter; +import org.dspace.sort.AbstractTextFilterOFD; + +/** + * Standard author ordering delegate implementation + * + * @author Graham Triggs + */ +public class OrderFormatAuthor extends AbstractTextFilterOFD +{ + { + filters = new TextFilter[] { new DecomposeDiactritics(), + new StripDiacritics(), + new LowerCaseAndTrim() }; + } +} diff --git a/dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatText.java b/dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatText.java new file mode 100644 index 0000000000..85b86a27cb --- /dev/null +++ b/dspace/modules/additions/src/main/java/org/dspace/sort/OrderFormatText.java @@ -0,0 +1,28 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.sort; + +import org.dspace.text.filter.DecomposeDiactritics; +import org.dspace.text.filter.StripDiacritics; +import org.dspace.text.filter.LowerCaseAndTrim; +import org.dspace.text.filter.TextFilter; +import org.dspace.sort.AbstractTextFilterOFD; + +/** + * Standard text ordering delegate implementation + * + * @author Graham Triggs + */ +public class OrderFormatText extends AbstractTextFilterOFD +{ + { + filters = new TextFilter[] { new DecomposeDiactritics(), + new StripDiacritics(), + new LowerCaseAndTrim() }; + } +} diff --git a/dspace/modules/xmlui/src/main/java/org/dspace/app/xmlui/aspect/artifactbrowser/ConfigurableBrowse.java b/dspace/modules/xmlui/src/main/java/org/dspace/app/xmlui/aspect/artifactbrowser/ConfigurableBrowse.java index cbae891f05..b467f7f2bf 100644 --- a/dspace/modules/xmlui/src/main/java/org/dspace/app/xmlui/aspect/artifactbrowser/ConfigurableBrowse.java +++ b/dspace/modules/xmlui/src/main/java/org/dspace/app/xmlui/aspect/artifactbrowser/ConfigurableBrowse.java @@ -24,9 +24,11 @@ import org.dspace.app.xmlui.wing.Message; import org.dspace.app.xmlui.wing.WingException; import org.dspace.app.xmlui.wing.element.*; +import org.dspace.app.xmlui.wing.element.List; import org.dspace.authorize.AuthorizeException; import org.dspace.browse.*; import org.dspace.content.*; +import org.dspace.content.Collection; import org.dspace.content.Item; import org.dspace.content.authority.factory.ContentAuthorityServiceFactory; import org.dspace.content.authority.service.ChoiceAuthorityService; @@ -41,10 +43,7 @@ import java.io.IOException; import java.io.Serializable; import java.sql.SQLException; -import java.util.HashMap; -import java.util.Locale; -import java.util.Map; -import java.util.Set; +import java.util.*; /** * Implements all the browse functionality (browse by title, subject, authors, @@ -1101,7 +1100,10 @@ Map getControlParameters() throws UIException paramMap.put(BrowseParams.RESULTS_PER_PAGE, Integer .toString(this.scope.getResultsPerPage())); paramMap.put(BrowseParams.ETAL, Integer.toString(this.etAl)); - + if (this.scope.hasStartsWith()) { + paramMap.put(BrowseParams.STARTS_WITH, this.scope.getStartsWith()); + } + return paramMap; } From 0045d37fc4a8188cd9fee420ef9bb5f307536fdb Mon Sep 17 00:00:00 2001 From: "Anne S. Lawrence" Date: Thu, 13 Jan 2022 15:12:42 -0500 Subject: [PATCH 16/23] remove Matomo CookieEomain statement --- .../src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl | 1 - 1 file changed, 1 deletion(-) diff --git a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl index e2369bd02b..fc56cebc60 100644 --- a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl +++ b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl @@ -337,7 +337,6 @@ var _paq = window._paq = window._paq || []; /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(["setDocumentTitle", document.domain + "/" + document.title]); - _paq.push(["setCookieDomain", "*.vtechworks.lib.vt.edu"]); _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { From 68ba5fd4e44c5f8335f8909965a43c8e73edbff7 Mon Sep 17 00:00:00 2001 From: alawvt Date: Mon, 12 Sep 2022 16:22:31 -0400 Subject: [PATCH 17/23] Revert irus (#814) * Revert "Add IRUS stats URL" This reverts commit 03b74cb7154bf84a0761a4acc310ae1040ab55b8. * Revert "Add IRUS patch (#766)" This reverts commit e51970723b021c41f2c764b52e6b6ce3f921cac7. --- dspace/config/dspace.cfg | 1 - dspace/config/hibernate.cfg.xml | 2 - dspace/config/launcher.xml | 7 - dspace/config/log4j.properties | 2 - dspace/config/modules/stats.cfg | 38 -- .../config/spring/api/core-dao-services.xml | 1 - .../spring/api/core-factory-services.xml | 1 - dspace/config/spring/api/core-services.xml | 2 - dspace/config/spring/api/discovery.xml | 2 - .../spring/jspui/open-url-listeners.xml | 12 - .../spring/xmlui/open-url-listeners.xml | 12 - dspace/modules/additions/pom.xml | 7 - .../atmire-statistics-exporter-api/pom.xml | 34 -- .../export/ExportUsageEventListener.java | 426 ------------------ .../statistics/export/OpenURLTracker.java | 82 ---- .../OpenURLTrackerLoggerServiceImpl.java | 34 -- .../export/RetryOpenUrlTracker.java | 63 --- .../export/dao/OpenURLTrackerDAO.java | 13 - .../dao/impl/OpenURLTrackerDAOImpl.java | 17 - .../OpenURLTrackerLoggerServiceFactory.java | 18 - ...penURLTrackerLoggerServiceFactoryImpl.java | 18 - .../service/OpenURLTrackerLoggerService.java | 19 - .../statistics/util/SpiderDetector.java | 263 ----------- .../usage/AbstractUsageEventListener.java | 70 --- .../V6.0_2017.02.14__statistics-harvester.sql | 27 -- .../V6.0_2017.02.14__statistics-harvester.sql | 8 - .../atmire-statistics-exporter/pom.xml | 24 - dspace/modules/jspui/pom.xml | 14 - dspace/modules/pom.xml | 1 - .../themes/vtmirage2/images/favicon_old.ico | Bin 22486 -> 0 bytes dspace/modules/xmlui/pom.xml | 8 - dspace/pom.xml | 38 +- dspace/src/main/config/build.xml | 22 - index.html | 1 - index.html.1 | 1 - 35 files changed, 1 insertion(+), 1287 deletions(-) delete mode 100644 dspace/config/modules/stats.cfg delete mode 100644 dspace/config/spring/jspui/open-url-listeners.xml delete mode 100644 dspace/config/spring/xmlui/open-url-listeners.xml delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql delete mode 100644 dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql delete mode 100644 dspace/modules/atmire-statistics-exporter/pom.xml delete mode 100644 dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/images/favicon_old.ico delete mode 100644 index.html delete mode 100644 index.html.1 diff --git a/dspace/config/dspace.cfg b/dspace/config/dspace.cfg index 909a12d51d..d17993d4d6 100644 --- a/dspace/config/dspace.cfg +++ b/dspace/config/dspace.cfg @@ -2025,4 +2025,3 @@ include = ${module_dir}/translator.cfg include = ${module_dir}/usage-statistics.cfg include = ${module_dir}/versioning.cfg include = ${module_dir}/workflow.cfg -include = ${module_dir}/stats.cfg \ No newline at end of file diff --git a/dspace/config/hibernate.cfg.xml b/dspace/config/hibernate.cfg.xml index 2f66f1891a..26595653d8 100644 --- a/dspace/config/hibernate.cfg.xml +++ b/dspace/config/hibernate.cfg.xml @@ -80,7 +80,5 @@ --> - - diff --git a/dspace/config/launcher.xml b/dspace/config/launcher.xml index 603bc6e229..138454c229 100644 --- a/dspace/config/launcher.xml +++ b/dspace/config/launcher.xml @@ -380,11 +380,4 @@ org.dspace.app.util.Version - - retry-tracker - Retry all failed commits to the OpenURLTracker - - com.atmire.statistics.export.RetryOpenUrlTracker - - diff --git a/dspace/config/log4j.properties b/dspace/config/log4j.properties index 8beeddc013..02231b3dbf 100644 --- a/dspace/config/log4j.properties +++ b/dspace/config/log4j.properties @@ -115,5 +115,3 @@ log4j.logger.org.dspace.services=ERROR log4j.logger.org.dspace.servicemanager=ERROR log4j.logger.org.dspace.providers=ERROR log4j.logger.org.dspace.utils=ERROR - -log4j.logger.com.atmire.statistics.export.ExportUsageEventListener=DEBUG \ No newline at end of file diff --git a/dspace/config/modules/stats.cfg b/dspace/config/modules/stats.cfg deleted file mode 100644 index 9312e85fec..0000000000 --- a/dspace/config/modules/stats.cfg +++ /dev/null @@ -1,38 +0,0 @@ - - -#-----------------------# -# Atmire stats exporter # -#-----------------------# - -# OPTIONAL metadata field used for filtering. -# If items with specific values for the "dc.type" field should be excluded, "dc.type" should be placed here. -# This should comply to the syntax schema.element.qualified or schema.element if the qualifier is null. -# stats.tracker.type-field = dc.type -# If "tracker.type-field" is set, the list of values must be defined in "tracker.type-value". -# This lists a comma separated list of values that will be excluded for the given field. -# stats.tracker.type-value = Article, Postprint - -# Enable the tracker -stats.tracker.enabled = true - -# Set the tracker environment to "test" or "production". Defaults to "test" if empty. -# The URL used by the test environment can be configured in property tracker.testurl -# The URL used by the production environment can be configured in property tracker.produrl -stats.tracker.environment = test -# The url used to test the submission of tracking info to. -stats.tracker.testurl = https://irus.jisc.ac.uk/counter/test/ -# The base url for submitting the tracking info to. -stats.tracker.produrl = https://irus.jisc.ac.uk/counter/us -# Identifies data as OpenURL 1.0 -stats.tracker.urlversion = Z39.88-2004 - -# The deployed user interface should be provided to build correct links to files. -# The dspace.type field can be set to either "xmlui" or "jspui". -stats.dspace.type = xmlui - -# Spider options -stats.spider.ipmatch.enabled = true -stats.spider.agentempty.enabled = false -stats.spider.agentregex.enabled = true -# Default is downloaded during build: ${dspace.dir}/config/COUNTER_Robots_list.txt -stats.spider.agentregex.regexfile = ${dspace.dir}/config/COUNTER_Robots_list.txt \ No newline at end of file diff --git a/dspace/config/spring/api/core-dao-services.xml b/dspace/config/spring/api/core-dao-services.xml index 85b8caf6c8..cc9244015e 100644 --- a/dspace/config/spring/api/core-dao-services.xml +++ b/dspace/config/spring/api/core-dao-services.xml @@ -57,7 +57,6 @@ - diff --git a/dspace/config/spring/api/core-factory-services.xml b/dspace/config/spring/api/core-factory-services.xml index ee15dcc2de..0078c02524 100644 --- a/dspace/config/spring/api/core-factory-services.xml +++ b/dspace/config/spring/api/core-factory-services.xml @@ -44,7 +44,6 @@ - diff --git a/dspace/config/spring/api/core-services.xml b/dspace/config/spring/api/core-services.xml index 26e38917c6..2c6160f55f 100644 --- a/dspace/config/spring/api/core-services.xml +++ b/dspace/config/spring/api/core-services.xml @@ -105,8 +105,6 @@ - - - - com.atmire - atmire-statistics-exporter-api - 1.0.0 - jar - - org.dspace dspace-api diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml deleted file mode 100644 index d4851a860b..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/pom.xml +++ /dev/null @@ -1,34 +0,0 @@ - - - 4.0.0 - - - com.atmire - atmire-statistics-exporter - 1.0.0 - - - com.atmire - atmire-statistics-exporter-api - 1.0.0 - jar - Atmire statistics exporter API - atmire.com - - - - org.dspace - dspace-api - - - - - javax.servlet - servlet-api - 2.3 - provided - - - diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java deleted file mode 100644 index c240afdf7e..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/ExportUsageEventListener.java +++ /dev/null @@ -1,426 +0,0 @@ -/* - * ExportUsageEventListener.java - * - * Version: $Revision: 1 $ - * Date: $Date: 2010-04-09 11:01:28 +0200 (vr, 09 apr 2010) $ - * Copyright (c) @mire. All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS - * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR - * TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE - * USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH - * DAMAGE. - */ -package com.atmire.statistics.export; - -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStreamReader; -import java.io.UnsupportedEncodingException; -import java.net.HttpURLConnection; -import java.net.URL; -import java.net.URLConnection; -import java.net.URLEncoder; -import java.sql.SQLException; -import java.util.ArrayList; -import java.util.Date; -import java.util.List; -import java.util.UUID; -import javax.servlet.http.HttpServletRequest; - -import com.atmire.statistics.export.factory.OpenURLTrackerLoggerServiceFactory; -import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; -import org.apache.commons.lang.StringUtils; -import org.apache.log4j.Logger; -import org.dspace.app.util.Util; -import org.dspace.content.Bitstream; -import org.dspace.content.Bundle; -import org.dspace.content.DCDate; -import org.dspace.content.Item; -import org.dspace.content.MetadataField; -import org.dspace.content.MetadataValue; -import org.dspace.content.factory.ContentServiceFactory; -import org.dspace.content.service.MetadataFieldService; -import org.dspace.core.Context; -import org.dspace.core.LogManager; -import org.dspace.services.ConfigurationService; -import org.dspace.services.factory.DSpaceServicesFactory; -import org.dspace.services.model.Event; -import org.dspace.statistics.util.SpiderDetector; -import org.dspace.usage.AbstractUsageEventListener; -import org.dspace.usage.UsageEvent; - -/** - * User: kevin (kevin at atmire.com) - * Date: 30-mrt-2010 - * Time: 16:37:56 - */ -public class ExportUsageEventListener extends AbstractUsageEventListener { - /* Log4j logger*/ - private static Logger log = Logger.getLogger(ExportUsageEventListener.class); - - /* The metadata field which is to be checked for */ - private static MetadataField trackerType; - - /* A list of values the type might have */ - private static List trackerValues; - - /* The base url of the tracker */ - private static String baseUrl; - - private static String trackerUrlVersion; - - private static final String ITEM_VIEW = "Investigation"; - private static final String BITSTREAM_DOWNLOAD = "Request"; - - private static ConfigurationService configurationService; - - - public void init(Context context) { - try { - if (configurationService == null) { - configurationService = DSpaceServicesFactory.getInstance().getConfigurationService(); - } - if (trackerType == null) { - trackerType = resolveConfigPropertyToMetadataField(context,"tracker.type-field"); - - String[] metadataValues = configurationService.getArrayProperty("stats.tracker.type-value"); - if (metadataValues.length > 0) { - trackerValues = new ArrayList<>(); - for (String metadataValue : metadataValues) { - trackerValues.add(metadataValue.toLowerCase()); - } - } else { - trackerValues = null; - } - - if(StringUtils.equals(configurationService.getProperty("stats.tracker.environment"), "production")){ - baseUrl = configurationService.getProperty("stats.tracker.produrl"); - } - else { - baseUrl = configurationService.getProperty("stats.tracker.testurl"); - } - - trackerUrlVersion = configurationService.getProperty("stats.tracker.urlversion"); - - - } - } catch (Exception e) { - log.error("Unknown error resolving configuration for the export usage event.", e); - trackerType = null; - trackerValues = null; - baseUrl = null; - trackerUrlVersion = null; - } - } - - public void receiveEvent(Event event) { - if (event instanceof UsageEvent) { - UsageEvent ue = (UsageEvent) event; - Context context = ue.getContext(); - init(context); - boolean irusEnabled = configurationService.getBooleanProperty("stats.tracker.enabled"); - if (irusEnabled) { - try { - //Check for item investigation - if (ue.getObject() instanceof Item) { - Item item = (Item) ue.getObject(); - if(item.isArchived() && !ContentServiceFactory.getInstance().getItemService().canEdit(context, item)) { - init(context); - - if (shouldProcessItem(item)) { - processItem(ue.getContext(), item, null, ue.getRequest(), ITEM_VIEW); - } - } - } - //Check for bitstream download - if (ue.getObject() instanceof Bitstream) { - Bitstream bit = (Bitstream) ue.getObject(); - //Check for an item - if (0 < bit.getBundles().size()) { - if (!SpiderDetector.isSpider(ue.getRequest())) { - Bundle bundle = bit.getBundles().get(0); - if (bundle.getName() == null || !bundle.getName().equals("ORIGINAL")) - return; - - if (0 < bundle.getItems().size()) { - Item item = bundle.getItems().get(0); - - if (item.isArchived() && !ContentServiceFactory.getInstance().getItemService() - .canEdit(context, item)) { - //Check if we have a valid type of item ! - if (shouldProcessItem(item)) { - processItem(ue.getContext(), item, bit, ue.getRequest(), BITSTREAM_DOWNLOAD); - } - } - } - } else { - log.info("Robot (" + ue.getRequest().getHeader("user-agent") + ") accessed " + bit - .getName() + "/" + bit.getSource()); - } - } - } - } catch (Exception e) { - UUID id; - id = ue.getObject().getID(); - - int type; - try { - type = ue.getObject().getType(); - } catch (Exception e1) { - type = -1; - } - log.error(LogManager.getHeader(ue.getContext(), "Error while processing export of use event", - "Id: " + id + " type: " + type), e); - e.printStackTrace(); - } - } - } - } - - private boolean shouldProcessItem(Item item) { - if (trackerType != null && trackerValues != null) { - List types = ContentServiceFactory.getInstance().getItemService().getMetadata(item, trackerType.getMetadataSchema().getName(), trackerType.getElement(), trackerType.getQualifier(), Item.ANY); - - if (!types.isEmpty()) { - //Find out if we have a type that needs to be excluded - for (MetadataValue type : types) { - if (trackerValues.contains(type.getValue().toLowerCase())) { - //We have found no type so process this item - return false; - } - } - return true; - } else { - // No types in this item, so not excluded - return true; - } - } else { - // No types to be excluded - return true; - } - } - - private void processItem(Context context, Item item, Bitstream bitstream, HttpServletRequest request, String eventType) throws IOException, SQLException { - //We have a valid url collect the rest of the data - String clientIP = request.getRemoteAddr(); - if (configurationService.getBooleanProperty("useProxies", false) && request.getHeader("X-Forwarded-For") != null) { - /* This header is a comma delimited list */ - for (String xfip : request.getHeader("X-Forwarded-For").split(",")) { - /* proxy itself will sometime populate this header with the same value in - remote address. ordering in spec is vague, we'll just take the last - not equal to the proxy - */ - if (!request.getHeader("X-Forwarded-For").contains(clientIP)) { - clientIP = xfip.trim(); - } - } - } - String clientUA = StringUtils.defaultIfBlank(request.getHeader("USER-AGENT"), ""); - String referer = StringUtils.defaultIfBlank(request.getHeader("referer"), ""); - - //Start adding our data - StringBuilder data = new StringBuilder(); - data.append(URLEncoder.encode("url_ver", "UTF-8") + "=" + URLEncoder.encode(trackerUrlVersion, "UTF-8")); - data.append("&").append(URLEncoder.encode("req_id", "UTF-8")).append("=").append( URLEncoder.encode(clientIP, "UTF-8")); - data.append("&").append(URLEncoder.encode("req_dat", "UTF-8")).append("=").append( URLEncoder.encode(clientUA, "UTF-8")); - data.append("&").append(URLEncoder.encode("rft.artnum", "UTF-8")).append("=").append( URLEncoder.encode("oai:" + configurationService.getProperty("dspace.hostname") + ":" + item.getHandle(), "UTF-8")); - data.append("&").append(URLEncoder.encode("rfr_dat", "UTF-8")).append("=").append( URLEncoder.encode(referer, "UTF-8")); - data.append("&").append(URLEncoder.encode("rfr_id", "UTF-8")).append("=").append( URLEncoder.encode(configurationService.getProperty("dspace.hostname"), "UTF-8")); - data.append("&").append(URLEncoder.encode("url_tim", "UTF-8")).append("=").append( URLEncoder.encode(new DCDate(new Date()).toString(), "UTF-8")); - - if (BITSTREAM_DOWNLOAD.equals(eventType)) { - String bitstreamInfo = getBitstreamInfo(item, bitstream); - data.append("&").append( URLEncoder.encode("svc_dat", "UTF-8")).append("=").append( URLEncoder.encode(bitstreamInfo, "UTF-8")); - data.append("&").append( URLEncoder.encode("rft_dat", "UTF-8")).append("=").append( URLEncoder.encode(BITSTREAM_DOWNLOAD, "UTF-8")); - } else if (ITEM_VIEW.equals(eventType)) { - String itemInfo = getItemInfo(item); - data.append("&").append( URLEncoder.encode("svc_dat", "UTF-8")).append("=").append( URLEncoder.encode(itemInfo, "UTF-8")); - data.append("&").append( URLEncoder.encode("rft_dat", "UTF-8")).append("=").append( URLEncoder.encode(ITEM_VIEW, "UTF-8")); - } - - processUrl(context, baseUrl + "?" + data.toString()); - - } - - private String getBitstreamInfo(final Item item, final Bitstream bitstream) { - //only for jsp ui - // http://demo.dspace.org/jspui/handle/10673/2235 - // http://demo.dspace.org/jspui/bitstream/10673/2235/1/Captura.JPG - // - - - //only fror xmlui - // http://demo.dspace.org/xmlui/handle/10673/2235 - // http://demo.dspace.org/xmlui/bitstream/handle/10673/2235/Captura.JPG?sequence=1 - // - - String uiType = configurationService.getProperty("stats.dspace.type"); - StringBuilder sb = new StringBuilder(configurationService.getProperty("dspace.url")); - if ("jspui".equals(uiType)) { - - sb.append("/bitstream/").append(item.getHandle()).append("/").append(bitstream.getSequenceID()); - - // If we can, append the pretty name of the bitstream to the URL - try { - if (bitstream.getName() != null) { - sb.append("/").append(Util.encodeBitstreamName(bitstream.getName(), "UTF-8")); - } - } catch (UnsupportedEncodingException uee) { - // just ignore it, we don't have to have a pretty - // name at the end of the URL because the sequence id will - // locate it. However it means that links in this file might - // not work.... - } - - - } else { //xmlui - - String identifier = null; - if (item != null && item.getHandle() != null) { - identifier = "handle/" + item.getHandle(); - } else if (item != null) { - identifier = "item/" + item.getID(); - } else { - identifier = "id/" + bitstream.getID(); - } - - - sb.append("/bitstream/").append(identifier).append("/"); - - // If we can, append the pretty name of the bitstream to the URL - try { - if (bitstream.getName() != null) { - sb.append(Util.encodeBitstreamName(bitstream.getName(), "UTF-8")); - } - } catch (UnsupportedEncodingException uee) { - // just ignore it, we don't have to have a pretty - // name at the end of the URL because the sequence id will - // locate it. However it means that links in this file might - // not work.... - } - - sb.append("?sequence=").append(bitstream.getSequenceID()); - } - return sb.toString(); - } - - private String getItemInfo(final Item item) { - StringBuilder sb = new StringBuilder(configurationService.getProperty("dspace.url")); - sb.append("/handle/").append(item.getHandle()); - - return sb.toString(); - } - - - private static void processUrl(Context c, String urlStr) throws IOException, SQLException { - log.debug("Prepared to send url to tracker URL: " + urlStr); - System.out.println(urlStr); - URLConnection conn; - - try { - // Send data - URL url = new URL(urlStr); - conn = url.openConnection(); - - // Get the response - BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream())); - while (rd.readLine() != null) ; - - rd.close(); - if (((HttpURLConnection) conn).getResponseCode() != 200) { - ExportUsageEventListener.logfailed(c, urlStr); - } else if (log.isDebugEnabled()) { - log.debug("Successfully posted " + urlStr + " on " + new Date()); - } - } catch (Exception e) { - log.error("Failed to send url to tracker URL: " + urlStr); - ExportUsageEventListener.logfailed(c, urlStr); - } - } - - private static void tryReprocessFailed(Context context, OpenURLTracker tracker) throws SQLException { - boolean success = false; - URLConnection conn; - try { - URL url = new URL(tracker.getUrl()); - conn = url.openConnection(); - BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream())); - while (rd.readLine() != null) ; - rd.close(); - if (((HttpURLConnection) conn).getResponseCode() == 200) { - success = true; - } - } catch (Exception e) { - success = false; - } finally { - if (success) { - OpenURLTrackerLoggerServiceFactory.getInstance().getOpenUrlTrackerLoggerService().remove(context, tracker); - // If the tracker was able to post successfully, we remove it from the database - log.info("Successfully posted " + tracker.getUrl() + " from " + tracker.getUploadDate()); - } else { - // Still no luck - write an error msg but keep the entry in the table for future executions - log.error("Failed attempt from " + tracker.getUrl() + " originating from " + tracker.getUploadDate()); - } - } - } - - public static void reprocessFailedQueue(Context context) throws SQLException { - Context c = new Context(); - OpenURLTrackerLoggerServiceFactory instance = OpenURLTrackerLoggerServiceFactory.getInstance(); - if(instance==null){ - log.error("Error retrieving the \"OpenURLTrackerLoggerServiceFactory\" instance, aborting the processing"); - return; - } - OpenURLTrackerLoggerService openUrlTrackerLoggerService = instance.getOpenUrlTrackerLoggerService(); - if(openUrlTrackerLoggerService==null){ - log.error("Error retrieving the \"openUrlTrackerLoggerService\" instance, aborting the processing"); - return; - } - List openURLTrackers = openUrlTrackerLoggerService.findAll(c); - for(OpenURLTracker openURLTracker : openURLTrackers){ - ExportUsageEventListener.tryReprocessFailed(context, openURLTracker) ; - } - - try { - c.abort(); - } catch (Exception ignored) { - } - } - - public static void logfailed(Context context, String url) throws SQLException { - Date now = new Date(); - if (url.equals("")) return; - OpenURLTrackerLoggerService service = OpenURLTrackerLoggerServiceFactory.getInstance().getOpenUrlTrackerLoggerService(); - OpenURLTracker tracker = service.create(context); - tracker.setUploadDate(now); - tracker.setUrl(url); - // TODO service tracker update - } - - private static MetadataField resolveConfigPropertyToMetadataField(Context context, String fieldName) throws SQLException { - String metadataField = configurationService.getProperty("stats." + fieldName); - if (metadataField != null && 0 < metadataField.trim().length()) { - metadataField = metadataField.trim(); - MetadataFieldService metadataFieldService = ContentServiceFactory.getInstance().getMetadataFieldService(); - return metadataFieldService.findByElement(context,metadataField.split("\\.")[0],metadataField.split("\\.")[1],metadataField.split("\\.").length == 2 ? null : metadataField.split("\\.")[2]); - } - return null; - } -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java deleted file mode 100644 index dbc7667fc5..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTracker.java +++ /dev/null @@ -1,82 +0,0 @@ -package com.atmire.statistics.export; - -import org.dspace.core.ReloadableEntity; -import org.hibernate.proxy.HibernateProxyHelper; - -import javax.persistence.*; -import java.util.Date; - -/** - * Created by jonas - jonas@atmire.com on 09/02/17. - */ -@Entity -@Table(name="OpenUrlTracker") -public class OpenURLTracker implements ReloadableEntity { - - @Id - @Column(name="tracker_id") - @GeneratedValue(strategy = GenerationType.SEQUENCE ,generator="openurltracker_seq") - @SequenceGenerator(name="openurltracker_seq", sequenceName="openurltracker_seq", allocationSize = 1) - private Integer id; - - @Column(name = "tracker_url", length = 1000) - private String url; - - @Column(name = "uploaddate") - @Temporal(TemporalType.DATE) - private Date uploadDate; - - protected OpenURLTracker(){ - - } - - @Override - public Integer getID() { - return id; - } - - - public String getUrl() { - return url; - } - - public void setUrl(String url) { - this.url = url; - } - - public Date getUploadDate() { - return uploadDate; - } - - public void setUploadDate(Date uploadDate) { - this.uploadDate = uploadDate; - } - - - @Override - public boolean equals(Object o) { - if (this == o) - { - return true; - } - Class objClass = HibernateProxyHelper.getClassWithoutInitializingProxy(o); - if (getClass() != objClass) - { - return false; - } - - final OpenURLTracker that = (OpenURLTracker)o; - if (this.getID() != that.getID()) - { - return false; - } - - return true; - } - @Override - public int hashCode() { - int hash=8; - hash=74*hash+ this.getID(); - return hash; - } -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java deleted file mode 100644 index 707d6fb703..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/OpenURLTrackerLoggerServiceImpl.java +++ /dev/null @@ -1,34 +0,0 @@ -package com.atmire.statistics.export; - -import com.atmire.statistics.export.dao.OpenURLTrackerDAO; -import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; -import org.dspace.core.Context; -import org.springframework.beans.factory.annotation.Autowired; - -import java.sql.SQLException; -import java.util.List; - -/** - * Created by jonas - jonas@atmire.com on 09/02/17. - */ -public class OpenURLTrackerLoggerServiceImpl implements OpenURLTrackerLoggerService { - - @Autowired(required = true) - protected OpenURLTrackerDAO openURLTrackerDAO; - - @Override - public void remove(Context context, OpenURLTracker openURLTracker) throws SQLException { - openURLTrackerDAO.delete(context,openURLTracker); - } - - @Override - public List findAll(Context context) throws SQLException { - return openURLTrackerDAO.findAll(context,OpenURLTracker.class); - } - - @Override - public OpenURLTracker create(Context context) throws SQLException { - OpenURLTracker openURLTracker = openURLTrackerDAO.create(context, new OpenURLTracker()); - return openURLTracker; - } -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java deleted file mode 100644 index 758c4d85a3..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/RetryOpenUrlTracker.java +++ /dev/null @@ -1,63 +0,0 @@ -package com.atmire.statistics.export; - -import org.apache.commons.cli.*; -import org.apache.log4j.Logger; -import org.dspace.core.Context; - -import java.sql.SQLException; - -/** - * Created by IntelliJ IDEA. - * Date: 25/04/12 - * Time: 09:32 - * - * @author: Kevin Van Ransbeeck (kevin van ransbeeck @ atmire dot com) - */ -public class RetryOpenUrlTracker { - private static final Logger log = Logger.getLogger(RetryOpenUrlTracker.class); - - /* Command Line execution */ - public static void main(String[] args) throws SQLException { - Context context = new Context(); - - context.turnOffAuthorisationSystem(); - - String usage = "com.atmire.statistics.export.RetryOpenUrlTracker [-a ]] or nothing to retry all failed attempts."; - Options options = new Options(); - HelpFormatter formatter = new HelpFormatter(); - CommandLine line = null; - - OptionBuilder.withArgName("Open URL Tracker"); - OptionBuilder.hasArg(true); - OptionBuilder.withDescription("Add a new row to the table (test purposes only)"); - options.addOption(OptionBuilder.create("a")); - OptionBuilder.isRequired(false); - OptionBuilder.withDescription("print this help message"); - options.addOption(OptionBuilder.create("h")); - - try { - line = new PosixParser().parse(options, args); - } catch (Exception e) { - // automatically generate the help statement - formatter.printHelp(usage, e.getMessage(), options, ""); - System.exit(1); - } - if (line.hasOption("h")) { - // automatically generate the help statement - formatter.printHelp(usage, options); - System.exit(1); - } - - if (line.hasOption("a")) { - ExportUsageEventListener.logfailed(context, line.getOptionValue("a")); - log.info("Created dummy entry in OpenUrlTracker with URL: " + line.getOptionValue("a")); - } else { - ExportUsageEventListener.reprocessFailedQueue(context); - } - context.restoreAuthSystemState(); - try { - context.complete(); - } catch (Exception ignored) { - } - } -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java deleted file mode 100644 index 6218f2bfe1..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/OpenURLTrackerDAO.java +++ /dev/null @@ -1,13 +0,0 @@ -package com.atmire.statistics.export.dao; - - -import com.atmire.statistics.export.OpenURLTracker; -import org.dspace.core.GenericDAO; - -/** - * Created by jonas - jonas@atmire.com on 09/02/17. - */ -public interface OpenURLTrackerDAO extends GenericDAO { - - -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java deleted file mode 100644 index 71619dbac4..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/dao/impl/OpenURLTrackerDAOImpl.java +++ /dev/null @@ -1,17 +0,0 @@ -package com.atmire.statistics.export.dao.impl; - -import com.atmire.statistics.export.OpenURLTracker; -import com.atmire.statistics.export.dao.OpenURLTrackerDAO; -import org.dspace.core.AbstractHibernateDAO; - -/** - * Created by jonas - jonas@atmire.com on 09/02/17. - */ -public class OpenURLTrackerDAOImpl extends AbstractHibernateDAO implements OpenURLTrackerDAO { - - - protected OpenURLTrackerDAOImpl(){ - super(); - } - -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java deleted file mode 100644 index 422aea6972..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactory.java +++ /dev/null @@ -1,18 +0,0 @@ -package com.atmire.statistics.export.factory; - -import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; -import org.dspace.services.factory.DSpaceServicesFactory; - -/** - * Created by jonas - jonas@atmire.com on 09/02/17. - */ -public abstract class OpenURLTrackerLoggerServiceFactory { - - public abstract OpenURLTrackerLoggerService getOpenUrlTrackerLoggerService(); - - public static OpenURLTrackerLoggerServiceFactory getInstance(){ - return DSpaceServicesFactory.getInstance().getServiceManager().getServiceByName("openURLTrackerLoggerServiceFactory", OpenURLTrackerLoggerServiceFactory.class); - - } - -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java deleted file mode 100644 index 48362f708e..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/factory/OpenURLTrackerLoggerServiceFactoryImpl.java +++ /dev/null @@ -1,18 +0,0 @@ -package com.atmire.statistics.export.factory; - -import com.atmire.statistics.export.service.OpenURLTrackerLoggerService; -import org.springframework.beans.factory.annotation.Autowired; - -/** - * Created by jonas - jonas@atmire.com on 09/02/17. - */ -public class OpenURLTrackerLoggerServiceFactoryImpl extends OpenURLTrackerLoggerServiceFactory{ - - @Autowired(required = true) - private OpenURLTrackerLoggerService openURLTrackerLoggerService; - - @Override - public OpenURLTrackerLoggerService getOpenUrlTrackerLoggerService() { - return openURLTrackerLoggerService; - } -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java deleted file mode 100644 index 8476618ccb..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/com/atmire/statistics/export/service/OpenURLTrackerLoggerService.java +++ /dev/null @@ -1,19 +0,0 @@ -package com.atmire.statistics.export.service; - -import com.atmire.statistics.export.OpenURLTracker; -import org.dspace.core.Context; - -import java.sql.SQLException; -import java.util.List; - -/** - * Created by jonas - jonas@atmire.com on 09/02/17. - */ -public interface OpenURLTrackerLoggerService { - - void remove(Context context, OpenURLTracker openURLTracker) throws SQLException; - - List findAll(Context context) throws SQLException; - - OpenURLTracker create(Context context) throws SQLException; -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java deleted file mode 100644 index b7ec11fbfa..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java +++ /dev/null @@ -1,263 +0,0 @@ -/** - * The contents of this file are subject to the license and copyright - * detailed in the LICENSE and NOTICE files at the root of the source - * tree and available online at - * - * http://www.dspace.org/license/ - */ -package org.dspace.statistics.util; - -import java.io.*; -import java.util.*; -import java.util.regex.*; -import javax.servlet.http.*; -import org.apache.log4j.*; -import org.dspace.services.factory.*; -import org.dspace.statistics.factory.*; - -/** - * SpiderDetector is used to find IP's that are spiders... - * In future someone may add UserAgents and Host Domains - * to the detection criteria here. - * - * @author kevinvandevelde at atmire.com - * @author ben at atmire.com - * @author Mark Diggory (mdiggory at atmire.com) - * @author Kevin Van Ransbeeck at atmire.com - */ -public class SpiderDetector { - - private static final Logger log = Logger.getLogger(SpiderDetector.class); - - - - /** - * Sparse HAshTable structure to hold IP Address Ranges. - */ - private static IPTable table = null; - private static Set spidersRegex = Collections.synchronizedSet(new HashSet()); - private static Set spidersMatched = null; - - /** - * Utility method which Reads the ip addresses out a file & returns them in a Set - * - * @param spiderIpFile the location of our spider file - * @return a vector full of ip's - * @throws java.io.IOException could not happen since we check the file be4 we use it - */ - public static Set readIpAddresses(File spiderIpFile) throws IOException { - Set ips = new HashSet<>(); - - if (!spiderIpFile.exists() || !spiderIpFile.isFile()) { - return ips; - } - - //Read our file & get all them ip's - try (BufferedReader in = new BufferedReader(new FileReader(spiderIpFile))){ - String line; - while ((line = in.readLine()) != null) { - if (!line.startsWith("#")) { - line = line.trim(); - - if (!line.equals("") && !Character.isDigit(line.charAt(0))) { - // is a hostname - // add this functionality later... - } else if (!line.equals("")) { - ips.add(line); - // is full v4 ip (too tired to deal with v6)... - } - } else { - // ua.add(line.replaceFirst("#","").replaceFirst("UA","").trim()); - // ... add this functionality later - } - } - } - return ips; - } - - /** - * Get an immutable Set representing all the Spider Addresses here - * - * @return Set setOfIpAddresses - */ - public static Set getSpiderIpAddresses() { - loadSpiderIpAddresses(); - return table.toSet(); - } - - /* - private loader to populate the table from files. - */ - - private static synchronized void loadSpiderIpAddresses() { - if (table == null) { - table = new IPTable(); - - String filePath = DSpaceServicesFactory.getInstance().getConfigurationService().getProperty("dspace.dir"); - try { - File spidersDir = new File(filePath, "config/spiders"); - - if (spidersDir.exists() && spidersDir.isDirectory()) { - for (File file : spidersDir.listFiles()) { - for (String ip : readIpAddresses(file)) { - table.add(ip); - } - log.info("Loaded Spider IP file: " + file); - } - } else { - log.info("No spider file loaded"); - } - } catch (Exception e) { - log.error("Error Loading Spiders:" + e.getMessage(), e); - } - } - } - - /** - * Static Service Method for testing spiders against existing spider files. - *

- * In the future this will be extended to support User Agent and - * domain Name detection. - *

- * In future spiders HashSet may be optimized as byte offset array to - * improve performance and memory footprint further. - * - * @param request - * @return true|false if the request was detected to be from a spider - */ - public static boolean isSpider(HttpServletRequest request) { - /* - * 1) If the IP address matches the spider IP addresses (this is the current implementation) - */ - boolean checkSpidersIP = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.ipmatch.enabled", true, true); - if (checkSpidersIP) { - if (StatisticsServiceFactory.getInstance().getSolrLoggerService().isUseProxies() && request.getHeader("X-Forwarded-For") != null) { - /* This header is a comma delimited list */ - for (String xfip : request.getHeader("X-Forwarded-For").split(",")) { - if (isSpider(xfip)) { - log.debug("spider.ipmatch"); - return true; - } - } - } else if (isSpider(request.getRemoteAddr())) { - log.debug("spider.ipmatch"); - return true; - } - } - /* - * 2) if the user-agent header is empty - DISABLED BY DEFAULT - - */ - boolean checkSpidersEmptyAgent = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.agentempty.enabled", false, true); - if (checkSpidersEmptyAgent) { - if (request.getHeader("user-agent") == null || request.getHeader("user-agent").length() == 0) { - log.debug("spider.agentempty"); - return true; - } - } - /* - * 3) if the user-agent corresponds to one of the regexes at http://www.projectcounter.org/r4/COUNTER_robot_txt_list_Jan_2011.txt - */ - boolean checkSpidersTxt = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.agentregex.enabled", true, true); - if (checkSpidersTxt) { - String userAgent = request.getHeader("user-agent"); - - if (userAgent != null && !userAgent.equals("")) { - return isSpiderRegex(userAgent); - } - } - return false; - } - - /** - * Check individual IP is a spider. - * - * @param ip - * @return if is spider IP - */ - public static boolean isSpider(String ip) { - if (table == null) { - SpiderDetector.loadSpiderIpAddresses(); - } - - try { - if (table.contains(ip)) { - return true; - } - } catch (Exception e) { - return false; - } - - return false; - } - - /** - * Checks the user-agent string vs a set of known regexes from spiders - * A second Set is kept for fast-matching. - * If a user-agent is matched once, it is added to this set with "known agents". - * If this user-agent comes back later, we can do a quick lookup in this set, - * instead of having to loop over the entire set with regexes again. - * - * @param userAgent String - * @return true if the user-agent matches a regex - */ - public static boolean isSpiderRegex(String userAgent) { - if (spidersMatched != null && spidersMatched.contains(userAgent)) { - log.debug("spider.agentregex"); - return true; - } else { - synchronized(spidersRegex) { - if (spidersRegex.isEmpty()) - loadSpiderRegexFromFile(); - } - - if (spidersRegex != null) { - for (Object regex : spidersRegex.toArray()) { - Matcher matcher = ((Pattern) regex).matcher(userAgent); - if (matcher.find()) { - if (spidersMatched == null) { - spidersMatched = new HashSet<>(); - } - if (spidersMatched.size() >= 100) { - spidersMatched.clear(); - } - spidersMatched.add(userAgent); - log.debug("spider.agentregex"); - return true; - } - } - } - return false; - } - } - - /** - * Populate static Set spidersRegex from local txt file. - * Original file downloaded from http://www.projectcounter.org/r4/COUNTER_robot_txt_list_Jan_2011.txt during build - */ - public static void loadSpiderRegexFromFile() { - String spidersTxt = DSpaceServicesFactory.getInstance().getConfigurationService().getPropertyAsType("stats.spider.agentregex.regexfile", String.class); - DataInputStream in = null; - try { - FileInputStream fstream = new FileInputStream(spidersTxt); - in = new DataInputStream(fstream); - BufferedReader br = new BufferedReader(new InputStreamReader(in)); - String strLine; - while ((strLine = br.readLine()) != null) { - spidersRegex.add(Pattern.compile(strLine, Pattern.CASE_INSENSITIVE)); - } - log.info("Loaded Spider Regex file: " + spidersTxt); - } catch (FileNotFoundException e) { - log.error("File with spiders regex not found @ " + spidersTxt); - } catch (IOException e) { - log.error("Could not read from file " + spidersTxt); - } finally { - try { - if (in != null) { - in.close(); - } - } catch (IOException e) { - log.error("Could not close file " + spidersTxt); - } - } - } -} \ No newline at end of file diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java deleted file mode 100644 index e6cb685fc8..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/java/org/dspace/usage/AbstractUsageEventListener.java +++ /dev/null @@ -1,70 +0,0 @@ -/** - * The contents of this file are subject to the license and copyright - * detailed in the LICENSE and NOTICE files at the root of the source - * tree and available online at - * - * http://www.dspace.org/license/ - */ -package org.dspace.usage; - -import org.dspace.services.EventService; -import org.dspace.services.model.EventListener; -import org.springframework.beans.BeansException; -import org.springframework.beans.factory.config.BeanPostProcessor; - -/** - * AbstractUsageEventListener is used as the base class for listening events running - * in the EventService. - * - * @author Mark Diggory (mdiggory at atmire.com) - * @version $Revision: $ - */ -public abstract class AbstractUsageEventListener implements EventListener, BeanPostProcessor { - - public AbstractUsageEventListener() { - super(); - } - - @Override - public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException { - return bean; - } - - @Override - public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException { - if(beanName.equals("org.dspace.services.EventService")) - { - setEventService((EventService) bean); - } - return bean; - } - - /** - * Empty String[] flags to have Listener - * consume any event name prefixes. - */ - public String[] getEventNamePrefixes() { - return new String[0]; - } - - /** - * Currently consumes events generated for - * all resources. - */ - public String getResourcePrefix() { - return null; - } - - public void setEventService(EventService service) { - if(service != null) - { - service.registerEventListener(this); - } - else - { - throw new IllegalStateException("EventService handed to Listener cannot be null"); - } - - } - -} diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql deleted file mode 100644 index dd0d67b187..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/oracle/V6.0_2017.02.14__statistics-harvester.sql +++ /dev/null @@ -1,27 +0,0 @@ --- CREATE SEQUENCE "openurltracker_seq" start with 1 increment by 1 nocache nocycle noorder; --- --- CREATE table "OpenUrlTracker" ( --- "tracker_id" NUMBER NOT NULL, --- "tracker_url" VARCHAR2(255) NOT NULL, --- "uploaddate" DATE, --- constraint "OpenUrlTracker_PK" primary key ("tracker_id") --- ); --- --- CREATE trigger "BI_OpenUrlTracker" --- before insert on "OpenUrlTracker" --- for each row --- begin --- if :NEW."tracker_id" is null then --- select "openurltracker_seq".nextval into :NEW."tracker_id" from dual; --- end if; --- end; - -CREATE SEQUENCE openurltracker_seq; - -CREATE TABLE OpenUrlTracker -( - tracker_id NUMBER, - tracker_url VARCHAR2(1000), - uploaddate DATE, - CONSTRAINT OpenUrlTracker_PK PRIMARY KEY (tracker_id) -); diff --git a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql b/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql deleted file mode 100644 index 21b9d7401d..0000000000 --- a/dspace/modules/atmire-statistics-exporter/atmire-statistics-exporter-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2017.02.14__statistics-harvester.sql +++ /dev/null @@ -1,8 +0,0 @@ -CREATE SEQUENCE openurltracker_seq; - -CREATE TABLE OpenUrlTracker -( - tracker_id INTEGER PRIMARY KEY, - tracker_url VARCHAR(1000), - uploaddate DATE -); diff --git a/dspace/modules/atmire-statistics-exporter/pom.xml b/dspace/modules/atmire-statistics-exporter/pom.xml deleted file mode 100644 index bcff958db7..0000000000 --- a/dspace/modules/atmire-statistics-exporter/pom.xml +++ /dev/null @@ -1,24 +0,0 @@ - - - 4.0.0 - - com.atmire - atmire-statistics-exporter - pom - atmire.com - Atmire statistics exporter - 1.0.0 - - - org.dspace - modules - [6.0,6.10] - - - - atmire-statistics-exporter-api - - - diff --git a/dspace/modules/jspui/pom.xml b/dspace/modules/jspui/pom.xml index 9f94969fb2..90a0103c26 100644 --- a/dspace/modules/jspui/pom.xml +++ b/dspace/modules/jspui/pom.xml @@ -89,13 +89,6 @@ org.dspace dspace-jspui war - - - - WEB-INF/classes/** - @@ -110,13 +103,6 @@ - - com.atmire - atmire-statistics-exporter-api - 1.0.0 - jar - - org.dspace.modules additions diff --git a/dspace/modules/pom.xml b/dspace/modules/pom.xml index 8406dab7ce..32668031ae 100644 --- a/dspace/modules/pom.xml +++ b/dspace/modules/pom.xml @@ -25,7 +25,6 @@ as a dependency in most other modules in [src]/dspace/modules --> additions - atmire-statistics-exporter - - com.atmire - atmire-statistics-exporter-api - 1.0.0 - jar - - org.dspace diff --git a/dspace/pom.xml b/dspace/pom.xml index de87f5fedd..5a8c643b98 100644 --- a/dspace/pom.xml +++ b/dspace/pom.xml @@ -40,7 +40,7 @@ modules - + - - ${basedir} - true - - config/default.context.xml - config/dspace.cfg - config/log4j-handle-plugin.properties - config/log4j.properties - config/modules/curate.cfg - config/modules/com.atmire.statistics.oai.cfg - config/modules/solr-statistics.cfg - - - - - ${basedir} - false - - config/default.context.xml - config/dspace.cfg - config/log4j-handle-plugin.properties - config/log4j.properties - config/modules/curate.cfg - config/modules/com.atmire.statistics.oai.cfg - config/modules/solr-statistics.cfg - - - @@ -198,12 +168,6 @@ - - - com.atmire - atmire-statistics-exporter-api - 1.0.0 - com.lyncode builder-commons diff --git a/dspace/src/main/config/build.xml b/dspace/src/main/config/build.xml index b4848ab168..b785818073 100644 --- a/dspace/src/main/config/build.xml +++ b/dspace/src/main/config/build.xml @@ -122,7 +122,6 @@ Common usage: - @@ -184,7 +183,6 @@ Common usage: - @@ -864,8 +862,6 @@ Common usage: - - ==================================================================== The DSpace code has been installed. @@ -951,24 +947,6 @@ You may manually install this file by following these steps: - - - Downloading: https://raw.githubusercontent.com/atmire/COUNTER-Robots/master/generated/COUNTER_Robots_list.txt - - - - - - - - - - - - - - - diff --git a/index.html b/index.html deleted file mode 100644 index 15f23b8419..0000000000 --- a/index.html +++ /dev/null @@ -1 +0,0 @@ -

IRUS Test Tracker

\ No newline at end of file diff --git a/index.html.1 b/index.html.1 deleted file mode 100644 index 5ecd536e90..0000000000 --- a/index.html.1 +++ /dev/null @@ -1 +0,0 @@ -

IRUS Tracker

\ No newline at end of file From 84781c5df5fefe10e2199cbd8b82242caf4c1317 Mon Sep 17 00:00:00 2001 From: alawvt Date: Mon, 19 Sep 2022 14:47:47 -0400 Subject: [PATCH 18/23] update homepage links (#817) --- dspace/config/news-xmlui.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dspace/config/news-xmlui.xml b/dspace/config/news-xmlui.xml index f105a9d770..5c86ff4c6d 100644 --- a/dspace/config/news-xmlui.xml +++ b/dspace/config/news-xmlui.xml @@ -3,13 +3,13 @@
VTechWorks -

VTechWorks provides global access to Virginia Tech scholarship. VTechWorks includes journal articles, books, theses, dissertations, conference papers, slide presentations, technical reports, working papers, administrative documents, videos, images, and more by faculty, students, and staff. Write vtechworks@vt.edu to get help adding your content to VTechWorks. Read an article about VTechWorks or visit the Open@VT blog to learn about current VTechWorks activities.

+

VTechWorks provides global access to Virginia Tech scholarship, and is the repository for the university's open access policy. VTechWorks includes journal articles, books, theses, dissertations, conference papers, slide presentations, technical reports, working papers, administrative documents, videos, images, and more by faculty, students, and staff. Write vtechworks@vt.edu to get help adding your content to VTechWorks. Read an article about VTechWorks or visit the Open@VT blog to learn about current VTechWorks activities.

Want to publish a standalone dataset? Visit Virginia Tech Data Repository.

Faculty can deposit items to VTechWorks from Elements (EFARs). Visit the Provost's Elements page to learn more and to log in to Elements.

-

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

+

Want to see historical data on VTechWorks usage and content? See our spreadsheet of VTechWorks Stats and a map of global usage.

From 45411c328b2ae9f1b84925b16d9eab4ded26faf3 Mon Sep 17 00:00:00 2001 From: Keith Gilbertson Date: Wed, 14 Dec 2022 12:59:10 -0500 Subject: [PATCH 19/23] Changes to news blurb on main page - Underline open access policy link - Update address of open@vt.edu blog --- dspace/config/news-xmlui.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dspace/config/news-xmlui.xml b/dspace/config/news-xmlui.xml index 5c86ff4c6d..b8b3f3f4bc 100644 --- a/dspace/config/news-xmlui.xml +++ b/dspace/config/news-xmlui.xml @@ -3,7 +3,7 @@
VTechWorks -

VTechWorks provides global access to Virginia Tech scholarship, and is the repository for the university's open access policy. VTechWorks includes journal articles, books, theses, dissertations, conference papers, slide presentations, technical reports, working papers, administrative documents, videos, images, and more by faculty, students, and staff. Write vtechworks@vt.edu to get help adding your content to VTechWorks. Read an article about VTechWorks or visit the Open@VT blog to learn about current VTechWorks activities.

+

VTechWorks provides global access to Virginia Tech scholarship, and is the repository for the university's open access policy. VTechWorks includes journal articles, books, theses, dissertations, conference papers, slide presentations, technical reports, working papers, administrative documents, videos, images, and more by faculty, students, and staff. Write vtechworks@vt.edu to get help adding your content to VTechWorks. Read an article about VTechWorks or visit the Open@VT blog to learn about current VTechWorks activities.

Want to publish a standalone dataset? Visit Virginia Tech Data Repository.

From 5e70e37367bcf2fdcae7470e635f36b1033684d9 Mon Sep 17 00:00:00 2001 From: Keith Gilbertson Date: Wed, 14 Dec 2022 13:04:47 -0500 Subject: [PATCH 20/23] Update removal policy link in footer The link now goes directly to the "Requesting that Material be Amended or Removed" section of our policies page. The anchor names on our policy page appear to have been generated automatically; I suspect that this link in the footer may need updating occassionally if the anchors in case the anchor names are automatically regenerated. --- .../main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl index fc56cebc60..b663d3c132 100644 --- a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl +++ b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl @@ -755,7 +755,7 @@

If you believe that any material in VTechWorks should be removed, please see our policy and procedure for - Requesting that Material be Amended or Removed. + Requesting that Material be Amended or Removed. All takedown requests will be promptly acknowledged and investigated.

From 1655cc657a2697662dc9cd0402c4c9232b8b71b9 Mon Sep 17 00:00:00 2001 From: KD Weeks Date: Mon, 8 May 2023 06:37:49 -0400 Subject: [PATCH 21/23] Update input-forms.xml (#833) --- dspace/config/input-forms.xml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/dspace/config/input-forms.xml b/dspace/config/input-forms.xml index 82c2a183f6..855bbe3880 100644 --- a/dspace/config/input-forms.xml +++ b/dspace/config/input-forms.xml @@ -1247,10 +1247,19 @@ Choose a discipline + + Agribusiness + Agribusiness + + + Applied Animal Behavior and Welfare + Applied Animal Behavior and Welfare + Applied Nutrition and Physical Activity Applied Nutrition and Physical Activity + Education Education From da81c69c051357b6f2663ef8cb06bc9ce8122505 Mon Sep 17 00:00:00 2001 From: keithgee Date: Fri, 30 Jun 2023 21:55:57 -0400 Subject: [PATCH 22/23] Update for Google Analytics 4 --- .../vtmirage2/xsl/core/page-structure.xsl | 30 +++++++------------ 1 file changed, 10 insertions(+), 20 deletions(-) diff --git a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl index b663d3c132..714e21758d 100644 --- a/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl +++ b/dspace/modules/xmlui-mirage2/src/main/webapp/themes/vtmirage2/xsl/core/page-structure.xsl @@ -7,7 +7,6 @@ http://www.dspace.org/license/ --> - - - - - - @@ -755,7 +736,7 @@

If you believe that any material in VTechWorks should be removed, please see our policy and procedure for - Requesting that Material be Amended or Removed. + Requesting that Material be Amended or Removed. All takedown requests will be promptly acknowledged and investigated.

@@ -898,6 +879,15 @@ + + - + - +