HDDS-13210. Add httpfs container to docker compose file. #46
Conversation
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @jojochuang for the patch.
docker-compose.yaml
Outdated
| CORE-SITE.XML_fs.defaultFS: "ofs://om" | ||
| CORE-SITE.XML_hadoop.proxyuser.hadoop.hosts: "*" | ||
| CORE-SITE.XML_hadoop.proxyuser.hadoop.groups: "*" |
There was a problem hiding this comment.
These need to be added in common-config. Otherwise I got the following error when trying to put a file:
httpfs-1 | 07:44:03,141 ERROR OzoneClientFactory:267 - Couldn't create RpcClient protocol exception:
httpfs-1 | org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hadoop is not allowed to impersonate hadoop
httpfs-1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1584)
...
httpfs-1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1880)
httpfs-1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:260)
...
httpfs-1 | at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:50)
httpfs-1 | at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:108)
httpfs-1 | at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:194)
...
httpfs-1 | at org.apache.ozone.lib.service.hadoop.FileSystemAccessService.createFileSystem(FileSystemAccessService.java:339)
...
httpfs-1 | at org.apache.ozone.fs.http.server.HttpFSServer.handleCreate(HttpFSServer.java:1229)
httpfs-1 | at org.apache.ozone.fs.http.server.HttpFSServer.put(HttpFSServer.java:901)
docker-compose.yaml
Outdated
| CORE-SITE.XML_fs.defaultFS: "ofs://om" | ||
| CORE-SITE.XML_hadoop.proxyuser.hadoop.hosts: "*" | ||
| CORE-SITE.XML_hadoop.proxyuser.hadoop.groups: "*" | ||
| OZONE-SITE.XML_hdds.scm.safemode.min.datanode: ${OZONE_SAFEMODE_MIN_DATANODES:-1} |
There was a problem hiding this comment.
hdds.scm.safemode.min.datanode is not needed for httpfs, can be removed.
docker-compose.yaml
Outdated
| httpfs: | ||
| <<: *image | ||
| ports: | ||
| - 14000:14000 |
There was a problem hiding this comment.
nit: the rest of this file has indentation of 3 spaces, please use the same for the new service.
|
Thanks @adoroszlai i had initially added them to common-config and then figured i should made it role specific. Also I'm wondering instead of putting it in common-config, how about moving the proxy user configs to OM and have fs.defaultFS set at httpfs? I tested and that also works for me. |
Yes, that's probably the most specific working config. |
|
Merged. Thanks @adoroszlai for review |
2 participants
The change was copied from https://github.com/apache/ozone/pull/8556/files and verified to work.